When MC create bars realtime -where is the time stamp from ?
When MC create bars realtime -where is the time stamp from ?
I am running some systems on MC with CQG as data feed. After some hours into the day i reload the data and buy and sell signals often change. I wonder how MC creates the bars - does it use my computer time to start a new 5 minute bar ? Or does it follow exchange time stamps ? How come that when i reload a lot of OHLC bars change ?
- furytrader
- Posts: 354
- Joined: 30 Jul 2010
- Location: Chicago, IL
- Has thanked: 155 times
- Been thanked: 217 times
Re: When MC create bars realtime -where is the time stamp fr
A few things to remember:
1) Generally speaking, MultiCharts merely acts as a processor of data received from your data source; consequently, there could be changes on your data source's side when prices are re-loaded that are responsible for the variations;
2) Depending on how your trading system works, the signals generated in real-time could differ from historical back-tested (reloaded) signals because of how MultiCharts must interpret the progression of price action intra-bar. This is the case when you don't build your historical price charts from tick data but instead used a time-compressed format (i.e., simple 5 minute bars with open, high, low and close).
For example, when a new 5 minute bar is created, let's say that the market (whatever market) opened at 100 and soon rallied to 105, getting you long at 105, before it dropped back down to 98, stopping you out at the low, and then rallying again and closing at 104. Yes, trading is frustrating.
If MultiCharts has to interpret how this same price bar progressed historically, it will assume that it opened at 100, traded down to 98 (because the low of 98 is closer to 100 than the high of 105), rallied up to 105 (getting you long) and then closed at 104. Instead of you being stopped out for a 7 point loss, it will show you long with a 1 point loss at the end of the 5 minute period.
It would completely miss the fact that you got stopped out intra-bar because it can't tell that the market first went up to 105 and then back down to 98 before it closed at 104.
The way to rectify this is to build your 5 minute bars with shorter-interval data (tick data, 1 minute bars, etc.) This will lead to greater accuracy when you reload your data.
1) Generally speaking, MultiCharts merely acts as a processor of data received from your data source; consequently, there could be changes on your data source's side when prices are re-loaded that are responsible for the variations;
2) Depending on how your trading system works, the signals generated in real-time could differ from historical back-tested (reloaded) signals because of how MultiCharts must interpret the progression of price action intra-bar. This is the case when you don't build your historical price charts from tick data but instead used a time-compressed format (i.e., simple 5 minute bars with open, high, low and close).
For example, when a new 5 minute bar is created, let's say that the market (whatever market) opened at 100 and soon rallied to 105, getting you long at 105, before it dropped back down to 98, stopping you out at the low, and then rallying again and closing at 104. Yes, trading is frustrating.
If MultiCharts has to interpret how this same price bar progressed historically, it will assume that it opened at 100, traded down to 98 (because the low of 98 is closer to 100 than the high of 105), rallied up to 105 (getting you long) and then closed at 104. Instead of you being stopped out for a 7 point loss, it will show you long with a 1 point loss at the end of the 5 minute period.
It would completely miss the fact that you got stopped out intra-bar because it can't tell that the market first went up to 105 and then back down to 98 before it closed at 104.
The way to rectify this is to build your 5 minute bars with shorter-interval data (tick data, 1 minute bars, etc.) This will lead to greater accuracy when you reload your data.
Re: When MC create bars realtime -where is the time stamp fr
I use 5 min bars and i make sure there is only 1 trade per 5 minute bar -- so the problem of intrabar buy and exit orders (which comes first etc.) does not affect me at all. For me its more the fact that the OHLC prices of 5 min bars get overwritten by data reloads from CQG my data provider.
It would be perfect if 10 seconds after a 5 min bar is finished MC would reload automatically without getting dissconnected from the broker in autottrading. Maybe a warning should pop up and say that position has changed and what i want to do with it. This helps because when you trade on 50 charts and you make a global reload you have to connect to broker again on each single chart. (Can i connect all charts to broker at once ?)
Reloads and appearing/dissappearing signals on the chart is a real issue for me and thats why I try to understand why the OHLC of 5 min bars differ so much when i reload . Why are the bars not correct right away ? It must be because the bars are built based on my computer clock rather than exchange time .... I am not sure and thats why I ask .
Thanks
It would be perfect if 10 seconds after a 5 min bar is finished MC would reload automatically without getting dissconnected from the broker in autottrading. Maybe a warning should pop up and say that position has changed and what i want to do with it. This helps because when you trade on 50 charts and you make a global reload you have to connect to broker again on each single chart. (Can i connect all charts to broker at once ?)
Reloads and appearing/dissappearing signals on the chart is a real issue for me and thats why I try to understand why the OHLC of 5 min bars differ so much when i reload . Why are the bars not correct right away ? It must be because the bars are built based on my computer clock rather than exchange time .... I am not sure and thats why I ask .
Thanks
- furytrader
- Posts: 354
- Joined: 30 Jul 2010
- Location: Chicago, IL
- Has thanked: 155 times
- Been thanked: 217 times
Re: When MC create bars realtime -where is the time stamp fr
I would try to capture the "before" and "after" shots for the OHLC chart and ask CQG about this. It shouldn't be changing that much.
Re: When MC create bars realtime -where is the time stamp fr
FuryTrader has made good points (like he makes always ). Most data providers have 2 servers: real time quote server and historical data server. There are differences between data stored on these 2 servers (I am not sure why such differences should exist). So, if you reloading your charts, you are getting data from historical data servers - which could lead to different OHLC values.
Another issue is timestamp, which is your original question. PC clock is used for timestamp data. I use Interactive brokers. Often I see my PC time is different from IB time by 2-3 seconds, sometimes more. The data that gets stored on my machine gets timestamped using my PC time. A quick way to prove this is to compare timestamps of ticks from MC (go to quote manager -> right click symbol -> edit data -> select tick and trade) and compare this with T&S of your data provider. You will notice that MC timestamps to miliseconds and it matches your PC clock exactly. However, timestamps in QM of MC will be different from timestamps of your data provider T&S window.
The best way to reduce this problem (not completely solve) is to keep your PC clock synchronized to web. Click on time in bottom right of your windows PC -> change date and time settings -> internet time -> change settings -> select one of the time servers from the drop down menu. Typically your PC time gets synced with internet time once a week. To change this setting to sync every hour, go to run -> regedit -> HKEY_Local_Key_Machine\System\CurrentControlSet\services\W32time\Timeproviders\NtpClient
Change the decimal value of SpecialPollInterval to 3600 (for sync interval of 1 hour). Do not put anything less than 3600, as syncing faster than 1 hr, might lead to a ban on your computer from timeserver and your internet time will not update.
Another issue is timestamp, which is your original question. PC clock is used for timestamp data. I use Interactive brokers. Often I see my PC time is different from IB time by 2-3 seconds, sometimes more. The data that gets stored on my machine gets timestamped using my PC time. A quick way to prove this is to compare timestamps of ticks from MC (go to quote manager -> right click symbol -> edit data -> select tick and trade) and compare this with T&S of your data provider. You will notice that MC timestamps to miliseconds and it matches your PC clock exactly. However, timestamps in QM of MC will be different from timestamps of your data provider T&S window.
The best way to reduce this problem (not completely solve) is to keep your PC clock synchronized to web. Click on time in bottom right of your windows PC -> change date and time settings -> internet time -> change settings -> select one of the time servers from the drop down menu. Typically your PC time gets synced with internet time once a week. To change this setting to sync every hour, go to run -> regedit -> HKEY_Local_Key_Machine\System\CurrentControlSet\services\W32time\Timeproviders\NtpClient
Change the decimal value of SpecialPollInterval to 3600 (for sync interval of 1 hour). Do not put anything less than 3600, as syncing faster than 1 hr, might lead to a ban on your computer from timeserver and your internet time will not update.
- Henry MultiСharts
- Posts: 9165
- Joined: 25 Aug 2011
- Has thanked: 1264 times
- Been thanked: 2957 times
Re: When MC create bars realtime -where is the time stamp fr
Hello amadeu1,
CQG historical timestamps are coming from the server, realtime timestamps are coming from the CQG API.
CQG historical timestamps are coming from the server, realtime timestamps are coming from the CQG API.
In order to have precise intra-bar price movement information instead of Intra-bar Price Movement Assumptions - please use Bar Magnifier: How Signals are Calculated2) Depending on how your trading system works, the signals generated in real-time could differ from historical back-tested (reloaded) signals because of how MultiCharts must interpret the progression of price action intra-bar. This is the case when you don't build your historical price charts from tick data but instead used a time-compressed format (i.e., simple 5 minute bars with open, high, low and close).
Re: When MC create bars realtime -where is the time stamp fr
There are 2 more additional small but important steps that I missed, if you are running Windows 7 or Windows Server 2008. (Not sure about Windows 8/8.1.).The best way to reduce this problem (not completely solve) is to keep your PC clock synchronized to web. Click on time in bottom right of your windows PC -> change date and time settings -> internet time -> change settings -> select one of the time servers from the drop down menu. Typically your PC time gets synced with internet time once a week. To change this setting to sync every hour, go to run -> regedit -> HKEY_Local_Key_Machine\System\CurrentControlSet\services\W32time\Timeproviders\NtpClient
Change the decimal value of SpecialPollInterval to 3600 (for sync interval of 1 hour). Do not put anything less than 3600, as syncing faster than 1 hr, might lead to a ban on your computer from timeserver and your internet time will not update.
For PC clock to synchronize with internet time, Windows Time service needs to be running.
1) Click on Run and type "Services.msc". A new window will open with all the services. If you go down the list, you will see Windows Time, set this to Start Automatic (Delayed Start). It might be originally set to Manual.
2) Open Task Scheduler. On the left pane, Go to Task Scheduler Library->Microsoft -> Windows -> Time Synchronize. Disable it on the right pane. Yes, disable it.
If you do not do above steps, after a reboot, Windows Time service will not start and as a result your time synchronization with internet time will fail. When Scheduled Task "Time Synchronize" in point 2) above is enabled what it does is that upon reboot, it starts Windows Time service and after a few seconds it stops Windows Time service. For synchronization with internet, Windows Time service needs to be always up and running. This is the reason we need to disable in point 2) above.
There are other ways to achieve this, but above is the easiest, it seems. It took time to figure this out and I had to spend 3 hours digging up this information from various sources on the internet.
Re: When MC create bars realtime -where is the time stamp fr
hilbert, thanks for posting this. I was unaware of the two extra steps you mentioned. I use Windows 8. Can you post any of the links you found in the course of your research. I will research the issue for Windows 8 and post back.There are 2 more additional small but important steps that I missed, if you are running Windows 7 or Windows Server 2008. (Not sure about Windows 8/8.1.).
There are other ways to achieve this, but above is the easiest, it seems. It took time to figure this out and I had to spend 3 hours digging up this information from various sources on the internet.
Re: When MC create bars realtime -where is the time stamp fr
1) https://social.technet.microsoft.com/Fo ... networkinghilbert, thanks for posting this. I was unaware of the two extra steps you mentioned. I use Windows 8. Can you post any of the links you found in the course of your research. I will research the issue for Windows 8 and post back.
2) http://technet.microsoft.com/en-us/libr ... s.10).aspx
3) http://translate.google.com/translate?h ... 2385818/ja
Link number 3 is most relevant (Method 3 in link 3 is what I have posted). Link number 2 gives detailed information about windows time service.
I suppose solutions in link 1 might also work, but method 3 in link 3 worked best and sounded easiest to me for Windows7. Cheers.
Re: When MC create bars realtime -where is the time stamp fr
The most easy way to make proper time sync is to use "Thinkman's" D4 time server. I'm using it on WinServer 2012R2 and was using it on WinServer 2008R2 and Win7. As long as it is working on WinServer 2012R2 it should work on Win8.1 as well.
Link : http://www.thinkman.com/dimension4/
Just my 2cents.
Regards.
Ben
Link : http://www.thinkman.com/dimension4/
Just my 2cents.
Regards.
Ben
Re: When MC create bars realtime -where is the time stamp fr
Thanks ben! Yes, I agree this is a good solution. I remember 3-5 years ago, Dimension was not updated for ages and there was a lot of ambiguity whether it works correctly with Windows Vista or not. That was what had forced me to come up with this long process of syncing PC time with internet.The most easy way to make proper time sync is to use "Thinkman's" D4 time server. I'm using it on WinServer 2012R2 and was using it on WinServer 2008R2 and Win7. As long as it is working on WinServer 2012R2 it should work on Win8.1 as well.
Link : http://www.thinkman.com/dimension4/
Now, since Dimension works for Win7, I am curious if we do we gain more reliable and more precise time using Dimension compared to using my long approach?
From Dimension webpage, http://www.thinkman.com/dimension4/default.htm
"At an interval you specify, Dimension 4 connects to one of these Internet Time Servers, which you get to choose from an exhaustive list built direclty into Dimension 4. The Time Server then sends the correct time back to your computer, where Dimension 4 uses sophisticated algorithms to correctly adjust your computer's clock to with-in a few milliseconds of the real time."
Would you happen to know what kind of sophisticated algorithms above refers to and if the net result is Dimension gives more precise and more reliable time compared to the longer method?
Re: When MC create bars realtime -where is the time stamp fr
I have no knowledge about the "sophisticated algorithms". I just could imagine that these algo's are trying to measure your time difference to the "main time server" implementing the latency from the place from where you are trying to adjust your time. I don't see anything from other time adjusting tools. Just an idea.Would you happen to know what kind of sophisticated algorithms above refers to and if the net result is Dimension gives more precise and more reliable time compared to the longer method?
I'm adjusting every minute from the main server and always see some microsecond adjustments every minute. Than I adjust the trading clients via NTP.
Regards.
Ben
Last edited by bensat on 04 Nov 2014, edited 1 time in total.
Re: When MC create bars realtime -where is the time stamp fr
Ben, do you see approximately consistent adjustments (say X microseconds) every minute? If you have statistics, I am curious about the coefficient of variation. Thanks.I'm adjusting every minute from the main server and always see some microsecond adjustments every minute. Than I adjust the trading clients via NLP.
Re: When MC create bars realtime -where is the time stamp fr
Ben, Thanks for your comments.I have no knowledge about the "sophisticated algorithms". I just could imagine that these algo's are trying to measure your time difference to the "main time server" implementing the latency from the place from where you are trying to adjust your time. I don't see anything from other time adjusting tools. Just an idea.
I'm adjusting every minute from the main server and always see some microsecond adjustments every minute. Than I adjust the trading clients via NTP.
Can you please paste screenshots of your configuration of Dimension. I would like to configure it the "right way". I want to see how many times an hour are your syncing it, and how adjustments happen every minute. Screenshots will be very helpful. Thanks a lot.
Re: When MC create bars realtime -where is the time stamp fr
I have no statistics at all. But just an example for the time sync between minutes.Ben, do you see approximately consistent adjustments (say X microseconds) every minute? If you have statistics, I am curious about the coefficient of variation. Thanks.
Example @6.59pm (Adjustment : 0.04sec)
Example @7.00pm (Adjustment : 0.03sec)
But I don't want to overestimate the importance for me, because I don't trade mechanically with MC. Just wanted to show you that there is constant time sync necessary.
Regards.
Ben
- Attachments
-
- TimeSync_700pm.png
- (183.6 KiB) Downloaded 2288 times
-
- TimeSync_659pm.png
- (192.59 KiB) Downloaded 2271 times
Re: When MC create bars realtime -where is the time stamp fr
Sorry, when I wrote my first answer you already posted your question. So here are my settings. Btw, statistics is available in the [History] on the main settings screen.Ben, Thanks for your comments.I have no knowledge about the "sophisticated algorithms". I just could imagine that these algo's are trying to measure your time difference to the "main time server" implementing the latency from the place from where you are trying to adjust your time. I don't see anything from other time adjusting tools. Just an idea.
I'm adjusting every minute from the main server and always see some microsecond adjustments every minute. Than I adjust the trading clients via NTP.
Can you please paste screenshots of your configuration of Dimension. I would like to configure it the "right way". I want to see how many times an hour are your syncing it, and how adjustments happen every minute. Screenshots will be very helpful. Thanks a lot.
Hope this helps you all.
Regards.
Ben
Re: When MC create bars realtime -where is the time stamp fr
Thanks Ben. A couple of points may be of interest:
1) Typical oscillator drift in PC motherboards is 20-50 ppm. The better ones are 10-20 ppm. If we assume worst case of 50 ppm, the PC clock will drift by 180 ms per hour. A good PC clock will drift by less than 100 ms per hour.
2) It is a bit surprising that your adjustments are of the order of 30 ms per minute which would seem to indicate a 500 ppm oscillator. This is perhaps not the PC clock drift but an artifact of other variability. There are a number of sources of variability in time synchronizing over the Internet which include the network path to the time server, the network stacks in both the client and server computers, and finally the operating system scheduler itself.
On a related but different topic, I would be interested in hearing about good apps for Internet connection uptime monitoring.
1) Typical oscillator drift in PC motherboards is 20-50 ppm. The better ones are 10-20 ppm. If we assume worst case of 50 ppm, the PC clock will drift by 180 ms per hour. A good PC clock will drift by less than 100 ms per hour.
2) It is a bit surprising that your adjustments are of the order of 30 ms per minute which would seem to indicate a 500 ppm oscillator. This is perhaps not the PC clock drift but an artifact of other variability. There are a number of sources of variability in time synchronizing over the Internet which include the network path to the time server, the network stacks in both the client and server computers, and finally the operating system scheduler itself.
On a related but different topic, I would be interested in hearing about good apps for Internet connection uptime monitoring.
Re: When MC create bars realtime -where is the time stamp fr
First of all, I think it's going way toooo much OT now. My drifts are positive and negative and not on a fixed ppm.Thanks Ben. A couple of points may be of interest:
1) Typical oscillator drift in PC motherboards is 20-50 ppm. The better ones are 10-20 ppm. If we assume worst case of 50 ppm, the PC clock will drift by 180 ms per hour. A good PC clock will drift by less than 100 ms per hour.
2) It is a bit surprising that your adjustments are of the order of 30 ms per minute which would seem to indicate a 500 ppm oscillator. This is perhaps not the PC clock drift but an artifact of other variability. There are a number of sources of variability in time synchronizing over the Internet which include the network path to the time server, the network stacks in both the client and server computers, and finally the operating system scheduler itself.
On a related but different topic, I would be interested in hearing about good apps for Internet connection uptime monitoring.
I'm not 100% with you, as in my experience there is no good or bad MB in terms of clock drift and tweaking it. There are so many side effects which have "serious" impact on the clock drift. In my opinion and from my experience it's not just the motherboard. Just think about different voltages while overclocking or in "sleepy hollows', different measurements and adjustments on different OS's (just take a look how Win adjusts compared to Linux) and were it gets rounded etc etc. What about BIOS versions, CMOS voltages and so on. Even between Windows OS's itself there is or can be a huge diff on the same hardware. So don't take it down just like you did.
Please don't get me wrong, but don't make these issues way too big. I think general sync is necessary but that's it. In my opinion the sync between the exchange server time and the time on your workstation is more important then other facts or at least I prefer it to do it on my trading/production environment on Linux. There is a sync service between exchange and customer. But not for retailers. The sync between our workstations and any time sync server is just a "workaround" and not seriously pro.
There is a tool out there to "tweak" the clock drift internally. But it's just a "tweak".
But always interesting to talk about.
Best Regards.
Ben
Re: When MC create bars realtime -where is the time stamp fr
Hey ben,In my opinion the sync between the exchange server time and the time on your workstation is more important then other facts. There is a sync service between exchange and customer. But not for MC users. The sync between our workstations and any time sync server is just a "workaround" and not seriously pro.
There is a tool out there to "tweak" the clock drift internally. But it's just a "tweak".
Which is this service you are talking about? When you say its not for MC users, are you hinting at that this service is very expensive and only hfts or big institutions have a need for this. Thanks.
Re: When MC create bars realtime -where is the time stamp fr
1) This article describes the support boundary to configure the Windows Time (W32Time) service for high-accuracy environments. High-accuracy environments include exchange trading houses and high-precision manufacturing companies for whose success critically accurate time measurement is required.Hey ben,
Which is this service you are talking about? When you say its not for MC users, are you hinting at that this service is very expensive and only hfts or big institutions have a need for this. Thanks.
http://support.microsoft.com/kb/939322
2) Following links are for some commercial softwares that maintain time much more accurately than W32tm.exe (which is Microsoft windows time service tool).
http://www.nist.gov/pml/div688/grp40/softwarelist.cfm
Re: When MC create bars realtime -where is the time stamp fr
What is it worth when you realize that Windows on it's own is able to adjust/sync into milliseconds only ? It can measure hectoseconds, but is not able to sync or adjust it. Sounds weird, but is fact.1) This article describes the support boundary to configure the Windows Time (W32Time) service for high-accuracy environments. High-accuracy environments include exchange trading houses and high-precision manufacturing companies for whose success critically accurate time measurement is required.Hey ben,
Which is this service you are talking about? When you say its not for MC users, are you hinting at that this service is very expensive and only hfts or big institutions have a need for this. Thanks.
http://support.microsoft.com/kb/939322
2) Following links are for some commercial softwares that maintain time much more accurately than W32tm.exe (which is Microsoft windows time service tool).
http://www.nist.gov/pml/div688/grp40/softwarelist.cfm
Regards.
Ben
Re: When MC create bars realtime -where is the time stamp fr
Hi Ben,
Thank you for pointing out Dimension 4.
I loaded it up and was pretty surprised at the scale of adjustments and so frequently. Maybe I have a duff (new) motherboard! So I logged onto CME site and came up with the following -
http://www.cmegroup.com/confluence/disp ... e+Protocol.
Looks like they recommend a triangular polling which can be done with NTP.
Cheers
Michael
Thank you for pointing out Dimension 4.
I loaded it up and was pretty surprised at the scale of adjustments and so frequently. Maybe I have a duff (new) motherboard! So I logged onto CME site and came up with the following -
http://www.cmegroup.com/confluence/disp ... e+Protocol.
Looks like they recommend a triangular polling which can be done with NTP.
Cheers
Michael