When MC create bars realtime -where is the time stamp from ?

Questions about MultiCharts and user contributed studies.
amadeu1
Posts: 67
Joined: 11 Feb 2008
Has thanked: 1 time
Been thanked: 6 times

When MC create bars realtime -where is the time stamp from ?

Postby amadeu1 » 17 Oct 2014

I am running some systems on MC with CQG as data feed. After some hours into the day i reload the data and buy and sell signals often change. I wonder how MC creates the bars - does it use my computer time to start a new 5 minute bar ? Or does it follow exchange time stamps ? How come that when i reload a lot of OHLC bars change ?

User avatar
MAtricks
Posts: 789
Joined: 09 Apr 2012
Has thanked: 286 times
Been thanked: 288 times

Re: When MC create bars realtime -where is the time stamp fr

Postby MAtricks » 17 Oct 2014

What are your Chart Type settings in Format Instrument -> Settings?

User avatar
furytrader
Posts: 354
Joined: 30 Jul 2010
Location: Chicago, IL
Has thanked: 155 times
Been thanked: 217 times

Re: When MC create bars realtime -where is the time stamp fr

Postby furytrader » 17 Oct 2014

A few things to remember:

1) Generally speaking, MultiCharts merely acts as a processor of data received from your data source; consequently, there could be changes on your data source's side when prices are re-loaded that are responsible for the variations;

2) Depending on how your trading system works, the signals generated in real-time could differ from historical back-tested (reloaded) signals because of how MultiCharts must interpret the progression of price action intra-bar. This is the case when you don't build your historical price charts from tick data but instead used a time-compressed format (i.e., simple 5 minute bars with open, high, low and close).

For example, when a new 5 minute bar is created, let's say that the market (whatever market) opened at 100 and soon rallied to 105, getting you long at 105, before it dropped back down to 98, stopping you out at the low, and then rallying again and closing at 104. Yes, trading is frustrating.

If MultiCharts has to interpret how this same price bar progressed historically, it will assume that it opened at 100, traded down to 98 (because the low of 98 is closer to 100 than the high of 105), rallied up to 105 (getting you long) and then closed at 104. Instead of you being stopped out for a 7 point loss, it will show you long with a 1 point loss at the end of the 5 minute period.

It would completely miss the fact that you got stopped out intra-bar because it can't tell that the market first went up to 105 and then back down to 98 before it closed at 104.

The way to rectify this is to build your 5 minute bars with shorter-interval data (tick data, 1 minute bars, etc.) This will lead to greater accuracy when you reload your data.

amadeu1
Posts: 67
Joined: 11 Feb 2008
Has thanked: 1 time
Been thanked: 6 times

Re: When MC create bars realtime -where is the time stamp fr

Postby amadeu1 » 17 Oct 2014

I use 5 min bars and i make sure there is only 1 trade per 5 minute bar -- so the problem of intrabar buy and exit orders (which comes first etc.) does not affect me at all. For me its more the fact that the OHLC prices of 5 min bars get overwritten by data reloads from CQG my data provider.

It would be perfect if 10 seconds after a 5 min bar is finished MC would reload automatically without getting dissconnected from the broker in autottrading. Maybe a warning should pop up and say that position has changed and what i want to do with it. This helps because when you trade on 50 charts and you make a global reload you have to connect to broker again on each single chart. (Can i connect all charts to broker at once ?)

Reloads and appearing/dissappearing signals on the chart is a real issue for me and thats why I try to understand why the OHLC of 5 min bars differ so much when i reload . Why are the bars not correct right away ? It must be because the bars are built based on my computer clock rather than exchange time .... I am not sure and thats why I ask .

Thanks

User avatar
furytrader
Posts: 354
Joined: 30 Jul 2010
Location: Chicago, IL
Has thanked: 155 times
Been thanked: 217 times

Re: When MC create bars realtime -where is the time stamp fr

Postby furytrader » 17 Oct 2014

I would try to capture the "before" and "after" shots for the OHLC chart and ask CQG about this. It shouldn't be changing that much.

hilbert
Posts: 224
Joined: 17 Aug 2011
Has thanked: 76 times
Been thanked: 64 times

Re: When MC create bars realtime -where is the time stamp fr

Postby hilbert » 17 Oct 2014

FuryTrader has made good points (like he makes always :)). Most data providers have 2 servers: real time quote server and historical data server. There are differences between data stored on these 2 servers (I am not sure why such differences should exist). So, if you reloading your charts, you are getting data from historical data servers - which could lead to different OHLC values.

Another issue is timestamp, which is your original question. PC clock is used for timestamp data. I use Interactive brokers. Often I see my PC time is different from IB time by 2-3 seconds, sometimes more. The data that gets stored on my machine gets timestamped using my PC time. A quick way to prove this is to compare timestamps of ticks from MC (go to quote manager -> right click symbol -> edit data -> select tick and trade) and compare this with T&S of your data provider. You will notice that MC timestamps to miliseconds and it matches your PC clock exactly. However, timestamps in QM of MC will be different from timestamps of your data provider T&S window.

The best way to reduce this problem (not completely solve) is to keep your PC clock synchronized to web. Click on time in bottom right of your windows PC -> change date and time settings -> internet time -> change settings -> select one of the time servers from the drop down menu. Typically your PC time gets synced with internet time once a week. To change this setting to sync every hour, go to run -> regedit -> HKEY_Local_Key_Machine\System\CurrentControlSet\services\W32time\Timeproviders\NtpClient

Change the decimal value of SpecialPollInterval to 3600 (for sync interval of 1 hour). Do not put anything less than 3600, as syncing faster than 1 hr, might lead to a ban on your computer from timeserver and your internet time will not update.

amadeu1
Posts: 67
Joined: 11 Feb 2008
Has thanked: 1 time
Been thanked: 6 times

Re: When MC create bars realtime -where is the time stamp fr

Postby amadeu1 » 17 Oct 2014

Hello Hilbert

great answer !! Thanks a lot for the help. This wont solve it completely but I give it a try and maybe it gets better that way.

User avatar
Henry MultiСharts
Posts: 9165
Joined: 25 Aug 2011
Has thanked: 1264 times
Been thanked: 2957 times

Re: When MC create bars realtime -where is the time stamp fr

Postby Henry MultiСharts » 21 Oct 2014

Hello amadeu1,

CQG historical timestamps are coming from the server, realtime timestamps are coming from the CQG API.
2) Depending on how your trading system works, the signals generated in real-time could differ from historical back-tested (reloaded) signals because of how MultiCharts must interpret the progression of price action intra-bar. This is the case when you don't build your historical price charts from tick data but instead used a time-compressed format (i.e., simple 5 minute bars with open, high, low and close).
In order to have precise intra-bar price movement information instead of Intra-bar Price Movement Assumptions - please use Bar Magnifier: How Signals are Calculated

hilbert
Posts: 224
Joined: 17 Aug 2011
Has thanked: 76 times
Been thanked: 64 times

Re: When MC create bars realtime -where is the time stamp fr

Postby hilbert » 03 Nov 2014

The best way to reduce this problem (not completely solve) is to keep your PC clock synchronized to web. Click on time in bottom right of your windows PC -> change date and time settings -> internet time -> change settings -> select one of the time servers from the drop down menu. Typically your PC time gets synced with internet time once a week. To change this setting to sync every hour, go to run -> regedit -> HKEY_Local_Key_Machine\System\CurrentControlSet\services\W32time\Timeproviders\NtpClient

Change the decimal value of SpecialPollInterval to 3600 (for sync interval of 1 hour). Do not put anything less than 3600, as syncing faster than 1 hr, might lead to a ban on your computer from timeserver and your internet time will not update.
There are 2 more additional small but important steps that I missed, if you are running Windows 7 or Windows Server 2008. (Not sure about Windows 8/8.1.).

For PC clock to synchronize with internet time, Windows Time service needs to be running.
1) Click on Run and type "Services.msc". A new window will open with all the services. If you go down the list, you will see Windows Time, set this to Start Automatic (Delayed Start). It might be originally set to Manual.
2) Open Task Scheduler. On the left pane, Go to Task Scheduler Library->Microsoft -> Windows -> Time Synchronize. Disable it on the right pane. Yes, disable it.

If you do not do above steps, after a reboot, Windows Time service will not start and as a result your time synchronization with internet time will fail. When Scheduled Task "Time Synchronize" in point 2) above is enabled what it does is that upon reboot, it starts Windows Time service and after a few seconds it stops Windows Time service. For synchronization with internet, Windows Time service needs to be always up and running. This is the reason we need to disable in point 2) above.

There are other ways to achieve this, but above is the easiest, it seems. It took time to figure this out and I had to spend 3 hours digging up this information from various sources on the internet.

orion
Posts: 250
Joined: 01 Oct 2014
Has thanked: 65 times
Been thanked: 104 times

Re: When MC create bars realtime -where is the time stamp fr

Postby orion » 03 Nov 2014

There are 2 more additional small but important steps that I missed, if you are running Windows 7 or Windows Server 2008. (Not sure about Windows 8/8.1.).

There are other ways to achieve this, but above is the easiest, it seems. It took time to figure this out and I had to spend 3 hours digging up this information from various sources on the internet.
hilbert, thanks for posting this. I was unaware of the two extra steps you mentioned. I use Windows 8. Can you post any of the links you found in the course of your research. I will research the issue for Windows 8 and post back.

hilbert
Posts: 224
Joined: 17 Aug 2011
Has thanked: 76 times
Been thanked: 64 times

Re: When MC create bars realtime -where is the time stamp fr

Postby hilbert » 04 Nov 2014

hilbert, thanks for posting this. I was unaware of the two extra steps you mentioned. I use Windows 8. Can you post any of the links you found in the course of your research. I will research the issue for Windows 8 and post back.
1) https://social.technet.microsoft.com/Fo ... networking

2) http://technet.microsoft.com/en-us/libr ... s.10).aspx

3) http://translate.google.com/translate?h ... 2385818/ja

Link number 3 is most relevant (Method 3 in link 3 is what I have posted). Link number 2 gives detailed information about windows time service.

I suppose solutions in link 1 might also work, but method 3 in link 3 worked best and sounded easiest to me for Windows7. Cheers.

User avatar
bensat
Posts: 331
Joined: 04 Oct 2014
Has thanked: 46 times
Been thanked: 104 times

Re: When MC create bars realtime -where is the time stamp fr

Postby bensat » 04 Nov 2014

The most easy way to make proper time sync is to use "Thinkman's" D4 time server. I'm using it on WinServer 2012R2 and was using it on WinServer 2008R2 and Win7. As long as it is working on WinServer 2012R2 it should work on Win8.1 as well.

Link : http://www.thinkman.com/dimension4/

Just my 2cents.

Regards.

Ben

hilbert
Posts: 224
Joined: 17 Aug 2011
Has thanked: 76 times
Been thanked: 64 times

Re: When MC create bars realtime -where is the time stamp fr

Postby hilbert » 04 Nov 2014

The most easy way to make proper time sync is to use "Thinkman's" D4 time server. I'm using it on WinServer 2012R2 and was using it on WinServer 2008R2 and Win7. As long as it is working on WinServer 2012R2 it should work on Win8.1 as well.

Link : http://www.thinkman.com/dimension4/
Thanks ben! Yes, I agree this is a good solution. I remember 3-5 years ago, Dimension was not updated for ages and there was a lot of ambiguity whether it works correctly with Windows Vista or not. That was what had forced me to come up with this long process of syncing PC time with internet.

Now, since Dimension works for Win7, I am curious if we do we gain more reliable and more precise time using Dimension compared to using my long approach?

From Dimension webpage, http://www.thinkman.com/dimension4/default.htm
"At an interval you specify, Dimension 4 connects to one of these Internet Time Servers, which you get to choose from an exhaustive list built direclty into Dimension 4. The Time Server then sends the correct time back to your computer, where Dimension 4 uses sophisticated algorithms to correctly adjust your computer's clock to with-in a few milliseconds of the real time."

Would you happen to know what kind of sophisticated algorithms above refers to and if the net result is Dimension gives more precise and more reliable time compared to the longer method?

User avatar
bensat
Posts: 331
Joined: 04 Oct 2014
Has thanked: 46 times
Been thanked: 104 times

Re: When MC create bars realtime -where is the time stamp fr

Postby bensat » 04 Nov 2014

Would you happen to know what kind of sophisticated algorithms above refers to and if the net result is Dimension gives more precise and more reliable time compared to the longer method?
I have no knowledge about the "sophisticated algorithms". I just could imagine that these algo's are trying to measure your time difference to the "main time server" implementing the latency from the place from where you are trying to adjust your time. I don't see anything from other time adjusting tools. Just an idea.

I'm adjusting every minute from the main server and always see some microsecond adjustments every minute. Than I adjust the trading clients via NTP.

Regards.

Ben
Last edited by bensat on 04 Nov 2014, edited 1 time in total.

orion
Posts: 250
Joined: 01 Oct 2014
Has thanked: 65 times
Been thanked: 104 times

Re: When MC create bars realtime -where is the time stamp fr

Postby orion » 04 Nov 2014

I'm adjusting every minute from the main server and always see some microsecond adjustments every minute. Than I adjust the trading clients via NLP.
Ben, do you see approximately consistent adjustments (say X microseconds) every minute? If you have statistics, I am curious about the coefficient of variation. Thanks.

hilbert
Posts: 224
Joined: 17 Aug 2011
Has thanked: 76 times
Been thanked: 64 times

Re: When MC create bars realtime -where is the time stamp fr

Postby hilbert » 04 Nov 2014

I have no knowledge about the "sophisticated algorithms". I just could imagine that these algo's are trying to measure your time difference to the "main time server" implementing the latency from the place from where you are trying to adjust your time. I don't see anything from other time adjusting tools. Just an idea.

I'm adjusting every minute from the main server and always see some microsecond adjustments every minute. Than I adjust the trading clients via NTP.
Ben, Thanks for your comments.

Can you please paste screenshots of your configuration of Dimension. I would like to configure it the "right way". I want to see how many times an hour are your syncing it, and how adjustments happen every minute. Screenshots will be very helpful. Thanks a lot.

User avatar
bensat
Posts: 331
Joined: 04 Oct 2014
Has thanked: 46 times
Been thanked: 104 times

Re: When MC create bars realtime -where is the time stamp fr

Postby bensat » 04 Nov 2014

Ben, do you see approximately consistent adjustments (say X microseconds) every minute? If you have statistics, I am curious about the coefficient of variation. Thanks.
I have no statistics at all. But just an example for the time sync between minutes.

Example @6.59pm (Adjustment : 0.04sec)

Image

Example @7.00pm (Adjustment : 0.03sec)

Image

But I don't want to overestimate the importance for me, because I don't trade mechanically with MC. Just wanted to show you that there is constant time sync necessary.

Regards.

Ben
Attachments
TimeSync_700pm.png
(183.6 KiB) Downloaded 2288 times
TimeSync_659pm.png
(192.59 KiB) Downloaded 2271 times

User avatar
bensat
Posts: 331
Joined: 04 Oct 2014
Has thanked: 46 times
Been thanked: 104 times

Re: When MC create bars realtime -where is the time stamp fr

Postby bensat » 04 Nov 2014

I have no knowledge about the "sophisticated algorithms". I just could imagine that these algo's are trying to measure your time difference to the "main time server" implementing the latency from the place from where you are trying to adjust your time. I don't see anything from other time adjusting tools. Just an idea.

I'm adjusting every minute from the main server and always see some microsecond adjustments every minute. Than I adjust the trading clients via NTP.
Ben, Thanks for your comments.

Can you please paste screenshots of your configuration of Dimension. I would like to configure it the "right way". I want to see how many times an hour are your syncing it, and how adjustments happen every minute. Screenshots will be very helpful. Thanks a lot.
Sorry, when I wrote my first answer you already posted your question. So here are my settings. Btw, statistics is available in the [History] on the main settings screen.

Image

Hope this helps you all.

Regards.

Ben

orion
Posts: 250
Joined: 01 Oct 2014
Has thanked: 65 times
Been thanked: 104 times

Re: When MC create bars realtime -where is the time stamp fr

Postby orion » 04 Nov 2014

Thanks Ben. A couple of points may be of interest:

1) Typical oscillator drift in PC motherboards is 20-50 ppm. The better ones are 10-20 ppm. If we assume worst case of 50 ppm, the PC clock will drift by 180 ms per hour. A good PC clock will drift by less than 100 ms per hour.

2) It is a bit surprising that your adjustments are of the order of 30 ms per minute which would seem to indicate a 500 ppm oscillator. This is perhaps not the PC clock drift but an artifact of other variability. There are a number of sources of variability in time synchronizing over the Internet which include the network path to the time server, the network stacks in both the client and server computers, and finally the operating system scheduler itself.

On a related but different topic, I would be interested in hearing about good apps for Internet connection uptime monitoring.

User avatar
bensat
Posts: 331
Joined: 04 Oct 2014
Has thanked: 46 times
Been thanked: 104 times

Re: When MC create bars realtime -where is the time stamp fr

Postby bensat » 05 Nov 2014

Thanks Ben. A couple of points may be of interest:

1) Typical oscillator drift in PC motherboards is 20-50 ppm. The better ones are 10-20 ppm. If we assume worst case of 50 ppm, the PC clock will drift by 180 ms per hour. A good PC clock will drift by less than 100 ms per hour.

2) It is a bit surprising that your adjustments are of the order of 30 ms per minute which would seem to indicate a 500 ppm oscillator. This is perhaps not the PC clock drift but an artifact of other variability. There are a number of sources of variability in time synchronizing over the Internet which include the network path to the time server, the network stacks in both the client and server computers, and finally the operating system scheduler itself.

On a related but different topic, I would be interested in hearing about good apps for Internet connection uptime monitoring.
First of all, I think it's going way toooo much OT now. My drifts are positive and negative and not on a fixed ppm.

I'm not 100% with you, as in my experience there is no good or bad MB in terms of clock drift and tweaking it. There are so many side effects which have "serious" impact on the clock drift. In my opinion and from my experience it's not just the motherboard. Just think about different voltages while overclocking or in "sleepy hollows', different measurements and adjustments on different OS's (just take a look how Win adjusts compared to Linux) and were it gets rounded etc etc. What about BIOS versions, CMOS voltages and so on. Even between Windows OS's itself there is or can be a huge diff on the same hardware. So don't take it down just like you did.

Please don't get me wrong, but don't make these issues way too big. I think general sync is necessary but that's it. In my opinion the sync between the exchange server time and the time on your workstation is more important then other facts or at least I prefer it to do it on my trading/production environment on Linux. There is a sync service between exchange and customer. But not for retailers. The sync between our workstations and any time sync server is just a "workaround" and not seriously pro.

There is a tool out there to "tweak" the clock drift internally. But it's just a "tweak".

But always interesting to talk about.

Best Regards.

Ben

hilbert
Posts: 224
Joined: 17 Aug 2011
Has thanked: 76 times
Been thanked: 64 times

Re: When MC create bars realtime -where is the time stamp fr

Postby hilbert » 05 Nov 2014

In my opinion the sync between the exchange server time and the time on your workstation is more important then other facts. There is a sync service between exchange and customer. But not for MC users. The sync between our workstations and any time sync server is just a "workaround" and not seriously pro.

There is a tool out there to "tweak" the clock drift internally. But it's just a "tweak".
Hey ben,

Which is this service you are talking about? When you say its not for MC users, are you hinting at that this service is very expensive and only hfts or big institutions have a need for this. Thanks.

hilbert
Posts: 224
Joined: 17 Aug 2011
Has thanked: 76 times
Been thanked: 64 times

Re: When MC create bars realtime -where is the time stamp fr

Postby hilbert » 05 Nov 2014

Hey ben,

Which is this service you are talking about? When you say its not for MC users, are you hinting at that this service is very expensive and only hfts or big institutions have a need for this. Thanks.
1) This article describes the support boundary to configure the Windows Time (W32Time) service for high-accuracy environments. High-accuracy environments include exchange trading houses and high-precision manufacturing companies for whose success critically accurate time measurement is required.
http://support.microsoft.com/kb/939322

2) Following links are for some commercial softwares that maintain time much more accurately than W32tm.exe (which is Microsoft windows time service tool).
http://www.nist.gov/pml/div688/grp40/softwarelist.cfm

User avatar
bensat
Posts: 331
Joined: 04 Oct 2014
Has thanked: 46 times
Been thanked: 104 times

Re: When MC create bars realtime -where is the time stamp fr

Postby bensat » 05 Nov 2014

Hey ben,

Which is this service you are talking about? When you say its not for MC users, are you hinting at that this service is very expensive and only hfts or big institutions have a need for this. Thanks.
1) This article describes the support boundary to configure the Windows Time (W32Time) service for high-accuracy environments. High-accuracy environments include exchange trading houses and high-precision manufacturing companies for whose success critically accurate time measurement is required.
http://support.microsoft.com/kb/939322

2) Following links are for some commercial softwares that maintain time much more accurately than W32tm.exe (which is Microsoft windows time service tool).
http://www.nist.gov/pml/div688/grp40/softwarelist.cfm
What is it worth when you realize that Windows on it's own is able to adjust/sync into milliseconds only ? It can measure hectoseconds, but is not able to sync or adjust it. Sounds weird, but is fact.

Regards.

Ben

TA100
Posts: 54
Joined: 10 Feb 2014
Has thanked: 39 times
Been thanked: 9 times

Re: When MC create bars realtime -where is the time stamp fr

Postby TA100 » 05 Nov 2014

Hi Ben,

Thank you for pointing out Dimension 4.

I loaded it up and was pretty surprised at the scale of adjustments and so frequently. Maybe I have a duff (new) motherboard! So I logged onto CME site and came up with the following -

http://www.cmegroup.com/confluence/disp ... e+Protocol.

Looks like they recommend a triangular polling which can be done with NTP.

Cheers

Michael


Return to “MultiCharts”