IB Real-Time Data server timestamps + new tws update query
IB Real-Time Data server timestamps + new tws update query
After updating to the latest IB tws today, I've noticed something odd with real-time updates. What I noticed is MC stopped collecting and displaying real-time data. So, I tried to run the previous version if tws but nothing came up - it appears to be broken. So, I rebooted my PC, then reset the IB settings. I had the IB -> Real-Time Data -> use server timestamps -> ticked, and the reset turned it off as I expected. Everything is working fine again. Then I turned the setting above back on and things went strange again on some but not all of the symbols. Turned it off and everything is fine. Anyone else notice something odd with the latest tws update? Could be just coincidence and might be something else but I've had the setting above turned on for months without any problem. It's possible I suppose that turning the setting on and off may confuse MC too. Doesn't explain the initial problem, so it's a mystery.
- TJ
- Posts: 7746
- Joined: 29 Aug 2006
- Location: Global Citizen
- Has thanked: 1033 times
- Been thanked: 2225 times
- Bruce DeVault
- Posts: 438
- Joined: 19 Jan 2010
- Location: Washington DC
- Been thanked: 2 times
- Contact:
I doubt the time zone makes any difference. It's been working fine for many months. I'll experiment some more on Monday. The other thing I've been doing is turning off the filter for bars with zero volume every day to permit the collection of the DOW cash index after it has closed. I then turn it back on. That might have disturbed things too. I should have requested the patch to 5.5 to avoid this some time ago but I thought I wait for 6.0 final to be released. Should be close so I won't bother to request the fix now.What time zone are you in? Could this be to do with some times of the day you are in a different date than IB is?
-
- Posts: 742
- Joined: 09 Apr 2010
- Location: Texas
- Has thanked: 483 times
- Been thanked: 274 times
- Contact:
I updated TWS last night and my ES contract was collecting intermittantly all day, I checked and the "use server timestamps" was checked.
My backup computer is running a 2nd copy of MC and the ES contract was ok, I checked and the "use server timestamps " was unchecked.
I'm not sure how to fix it other than unchecking the "timestamp" option. I may have to overwrite the data with my backup data and backfill the missing data.
My backup computer is running a 2nd copy of MC and the ES contract was ok, I checked and the "use server timestamps " was unchecked.
I'm not sure how to fix it other than unchecking the "timestamp" option. I may have to overwrite the data with my backup data and backfill the missing data.
Me too. I haven't updated the other tws yet so before I do I will use both old and new, and compare the results.** I usually run new versions of TWS on my backup machine for a week or so, to be sure it's stable, this time unfortunately, I didn't.OK, that makes two of us. Not enough to prove it IB's latest upgrade of tws is the cause but sure looks that way.
-
- Posts: 742
- Joined: 09 Apr 2010
- Location: Texas
- Has thanked: 483 times
- Been thanked: 274 times
- Contact:
It was weird for me as ONLY the ES symbol was effected (futures and continuous contracts). I deleted all of yesterday's ES data (bid-ask-trade data) and cleared the ES cache, UNCHECKED the server data option, then let MC backfill and now the data is ok. I haven't tried it in RT though, Sunday night will be the final test.
-
- Posts: 742
- Joined: 09 Apr 2010
- Location: Texas
- Has thanked: 483 times
- Been thanked: 274 times
- Contact:
It was weird for me as ONLY the ES symbol was effected (futures and continuous contracts). I deleted all of yesterday's ES data (bid-ask-trade data) and cleared the ES cache, UNCHECKED the server data option, then let MC backfill and now the data is ok. I haven't tried it in RT though, Sunday night will be the final test.
*** Turns out my database was corrupted and I had to replace it with a backup from last week, took about 6 restarts of TWS to get all of the history back, because of the irritating TWS pacing violation errors.
Everything seems to be ok now. I still believe the TWS upgrade was at fault, as I had no problems for months before that. (due to checking the data server option)
I tried the older version of tws on my second PC and it worked fine. I then upgraded it and it started to misbehave. It proves IB have changed something. I don't understand why or how. In any case, I won't be using server timestamps ever again. It's a worry though that IB can do this. It's also odd that they can do something like this yet they refuse to fix the problem where historical volume is wrong while real-time volume is OK.*** Turns out my database was corrupted and I had to replace it with a backup from last week, took about 6 restarts of TWS to get all of the history back, because of the irritating TWS pacing violation errors.
Everything seems to be ok now. I still believe the TWS upgrade was at fault, as I had no problems for months before that. (due to checking the data server option)
-
- Posts: 742
- Joined: 09 Apr 2010
- Location: Texas
- Has thanked: 483 times
- Been thanked: 274 times
- Contact:
My backup computer stopped updating ES mid-morning today- I shut it down and went back to the previous version of TWS. I'm going to do the same on my main computer tonight. So far it is working ok on my main computer, but I don't trust the new TWS, they changed something that MC doesn't like. Beware of the current TWS version- 904.
- Bruce DeVault
- Posts: 438
- Joined: 19 Jan 2010
- Location: Washington DC
- Been thanked: 2 times
- Contact:
If you suspect it may have something to do with a TWS setting that's changed, it's easy to specify a new settings directory e.g. C:\JTS2 instead of C:\JTS to force TWS to start completely anew (you can always switch back to your previous settings) - this will allow you to easily check if you think something's gone amiss because of trying the gateway. (I don't think that SHOULD be the case, but I am very interested to hear your experiences with this so please do let us know how it works out.)
So far I'm not having any problems since switching off "use server timestamps". If I do have problems again I will try your suggestion (although just renaming the jts.ini file has worked for me in the past with other problems - for example I tried the VPN option to play with but tws stopped working even when I went back to the normal connection method).
-
- Posts: 742
- Joined: 09 Apr 2010
- Location: Texas
- Has thanked: 483 times
- Been thanked: 274 times
- Contact:
** I DID try the gateway over the weekend(MC wouldn't connect to it, even after changing the port), I didn't think it could be related, now I wonder.If you suspect it may have something to do with a TWS setting that's changed, it's easy to specify a new settings directory e.g. C:\JTS2 instead of C:\JTS to force TWS to start completely anew (you can always switch back to your previous settings) - this will allow you to easily check if you think something's gone amiss because of trying the gateway. (I don't think that SHOULD be the case, but I am very interested to hear your experiences with this so please do let us know how it works out.)
Mine did work but as I posted before I prefer the full tws. Now I will rename the jts.ini and try again with "use server timestamps turned" on to see how it goes.** I DID try the gateway over the weekend(MC wouldn't connect to it, even after changing the port), I didn't think it could be related, now I wonder.
-
- Posts: 742
- Joined: 09 Apr 2010
- Location: Texas
- Has thanked: 483 times
- Been thanked: 274 times
- Contact:
Using a fresh copy of tws working directory didn't help. Still have the problem with "use server timestamps" turned on. Now turned if off and all OK again. So, it appears attempting to use the IB gateway has nothing to do with it.
** Good , thanks for the update, I'm still going back to the previous version of TWS though, it ran flawlessly for many weeks. (I'll probably rename the tws dir too)
I could well be off-topic here but since time-stamps are all about data consistency across multiple clients (or even between client-server), I ll say it anyway.
I noticed long ago that the feed from IB as it is registered by MC could have a lot of minor differences among different clients. Even clients connected to the same IB server. At first I thought it was an IB problem but then I found out it was much simpler.
The minor differences in OHLC data is a result of not synchronized PC time.
After installing a time synchronizing service everything fell into place. Now the data bars are identical among different clients and almost identical to IB charts as well.
I guess not many people run multiple clients but for those who do, I totally recommend installing a time-synchronizing service like this one
http://www.meinberg.de/english/info/ntp.htm
I noticed long ago that the feed from IB as it is registered by MC could have a lot of minor differences among different clients. Even clients connected to the same IB server. At first I thought it was an IB problem but then I found out it was much simpler.
The minor differences in OHLC data is a result of not synchronized PC time.
After installing a time synchronizing service everything fell into place. Now the data bars are identical among different clients and almost identical to IB charts as well.
I guess not many people run multiple clients but for those who do, I totally recommend installing a time-synchronizing service like this one
http://www.meinberg.de/english/info/ntp.htm
- Bruce DeVault
- Posts: 438
- Joined: 19 Jan 2010
- Location: Washington DC
- Been thanked: 2 times
- Contact:
In real-time, IB does not supply timestamps with its "snapshot ticks" - as a consequence, the PC time is used in this case. This is a situation that's not true of most data feeds, it's an IB limitation.
It's actually documented to be this way in the MultiCharts help:
You should also be aware that latency between IB and you affects the bars that are built, so if your PC is located far from IB on the internet (as measured in "hops" or milliseconds) you may have less reliability in the bars that are built than others have.
It's actually documented to be this way in the MultiCharts help:
You're absolutely right, if you're using IB as a data source, you should make sure your time is synchronized and correct, to both minimize error relative to the time stamps at their end, and also maximize reproducibility on other machines.Interactive Brokers does not provide timestamps for quotes. Because of this limitation the program has to assign the system time to every quote. Such a workaround may bias the true timestamp of the quote by the time difference between its sending and receiving; in most cases the delay is fractions of a second
You should also be aware that latency between IB and you affects the bars that are built, so if your PC is located far from IB on the internet (as measured in "hops" or milliseconds) you may have less reliability in the bars that are built than others have.