IB Real-Time Data server timestamps + new tws update query

Questions about MultiCharts and user contributed studies.
janus
Posts: 755
Joined: 25 May 2009
Has thanked: 41 times
Been thanked: 84 times

IB Real-Time Data server timestamps + new tws update query

Postby janus » 15 Apr 2010

After updating to the latest IB tws today, I've noticed something odd with real-time updates. What I noticed is MC stopped collecting and displaying real-time data. So, I tried to run the previous version if tws but nothing came up - it appears to be broken. So, I rebooted my PC, then reset the IB settings. I had the IB -> Real-Time Data -> use server timestamps -> ticked, and the reset turned it off as I expected. Everything is working fine again. Then I turned the setting above back on and things went strange again on some but not all of the symbols. Turned it off and everything is fine. Anyone else notice something odd with the latest tws update? Could be just coincidence and might be something else but I've had the setting above turned on for months without any problem. It's possible I suppose that turning the setting on and off may confuse MC too. Doesn't explain the initial problem, so it's a mystery.

User avatar
TJ
Posts: 6601
Joined: 29 Aug 2006
Location: Global Citizen
Has thanked: 971 times
Been thanked: 1910 times

Postby TJ » 15 Apr 2010

for IB data feed,
use server timestamps -> should be unchecked.

janus
Posts: 755
Joined: 25 May 2009
Has thanked: 41 times
Been thanked: 84 times

Postby janus » 15 Apr 2010

for IB data feed,
use server timestamps -> should be unchecked.


Why? I've been told by TSS I could turn it on to try and fix another issue I had (but didn't help).

User avatar
Bruce DeVault
Posts: 438
Joined: 19 Jan 2010
Location: Washington DC
Been thanked: 2 times
Contact:

Postby Bruce DeVault » 16 Apr 2010

What time zone are you in? Could this be to do with some times of the day you are in a different date than IB is?

janus
Posts: 755
Joined: 25 May 2009
Has thanked: 41 times
Been thanked: 84 times

Postby janus » 16 Apr 2010

What time zone are you in? Could this be to do with some times of the day you are in a different date than IB is?


I doubt the time zone makes any difference. It's been working fine for many months. I'll experiment some more on Monday. The other thing I've been doing is turning off the filter for bars with zero volume every day to permit the collection of the DOW cash index after it has closed. I then turn it back on. That might have disturbed things too. I should have requested the patch to 5.5 to avoid this some time ago but I thought I wait for 6.0 final to be released. Should be close so I won't bother to request the fix now.

sptrader
Posts: 640
Joined: 09 Apr 2010
Location: Colorado
Has thanked: 403 times
Been thanked: 239 times
Contact:

Postby sptrader » 16 Apr 2010

I updated TWS last night and my ES contract was collecting intermittantly all day, I checked and the "use server timestamps" was checked.

My backup computer is running a 2nd copy of MC and the ES contract was ok, I checked and the "use server timestamps " was unchecked.

I'm not sure how to fix it other than unchecking the "timestamp" option. I may have to overwrite the data with my backup data and backfill the missing data.

janus
Posts: 755
Joined: 25 May 2009
Has thanked: 41 times
Been thanked: 84 times

Postby janus » 16 Apr 2010

OK, that makes two of us. Not enough to prove it IB's latest upgrade of tws is the cause but sure looks that way.

sptrader
Posts: 640
Joined: 09 Apr 2010
Location: Colorado
Has thanked: 403 times
Been thanked: 239 times
Contact:

Postby sptrader » 16 Apr 2010

OK, that makes two of us. Not enough to prove it IB's latest upgrade of tws is the cause but sure looks that way.


** I usually run new versions of TWS on my backup machine for a week or so, to be sure it's stable, this time unfortunately, I didn't.

Spaceant
Posts: 252
Joined: 30 May 2009
Has thanked: 1 time
Been thanked: 3 times

Postby Spaceant » 17 Apr 2010

I remember I read something that IB wouldn't time stamp the realtime quotes but historical ones.

If that is true, should we uncheck the box "use server timestamps"?

Sa

janus
Posts: 755
Joined: 25 May 2009
Has thanked: 41 times
Been thanked: 84 times

Postby janus » 17 Apr 2010

OK, that makes two of us. Not enough to prove it IB's latest upgrade of tws is the cause but sure looks that way.


** I usually run new versions of TWS on my backup machine for a week or so, to be sure it's stable, this time unfortunately, I didn't.


Me too. I haven't updated the other tws yet so before I do I will use both old and new, and compare the results.

sptrader
Posts: 640
Joined: 09 Apr 2010
Location: Colorado
Has thanked: 403 times
Been thanked: 239 times
Contact:

Postby sptrader » 17 Apr 2010

It was weird for me as ONLY the ES symbol was effected (futures and continuous contracts). I deleted all of yesterday's ES data (bid-ask-trade data) and cleared the ES cache, UNCHECKED the server data option, then let MC backfill and now the data is ok. I haven't tried it in RT though, Sunday night will be the final test.

sptrader
Posts: 640
Joined: 09 Apr 2010
Location: Colorado
Has thanked: 403 times
Been thanked: 239 times
Contact:

Postby sptrader » 18 Apr 2010

It was weird for me as ONLY the ES symbol was effected (futures and continuous contracts). I deleted all of yesterday's ES data (bid-ask-trade data) and cleared the ES cache, UNCHECKED the server data option, then let MC backfill and now the data is ok. I haven't tried it in RT though, Sunday night will be the final test.



*** Turns out my database was corrupted and I had to replace it with a backup from last week, took about 6 restarts of TWS to get all of the history back, because of the irritating TWS pacing violation errors.
Everything seems to be ok now. I still believe the TWS upgrade was at fault, as I had no problems for months before that. (due to checking the data server option)

janus
Posts: 755
Joined: 25 May 2009
Has thanked: 41 times
Been thanked: 84 times

Postby janus » 18 Apr 2010

*** Turns out my database was corrupted and I had to replace it with a backup from last week, took about 6 restarts of TWS to get all of the history back, because of the irritating TWS pacing violation errors.
Everything seems to be ok now. I still believe the TWS upgrade was at fault, as I had no problems for months before that. (due to checking the data server option)


I tried the older version of tws on my second PC and it worked fine. I then upgraded it and it started to misbehave. It proves IB have changed something. I don't understand why or how. In any case, I won't be using server timestamps ever again. It's a worry though that IB can do this. It's also odd that they can do something like this yet they refuse to fix the problem where historical volume is wrong while real-time volume is OK.

sptrader
Posts: 640
Joined: 09 Apr 2010
Location: Colorado
Has thanked: 403 times
Been thanked: 239 times
Contact:

Postby sptrader » 19 Apr 2010

My backup computer stopped updating ES mid-morning today- I shut it down and went back to the previous version of TWS. I'm going to do the same on my main computer tonight. So far it is working ok on my main computer, but I don't trust the new TWS, they changed something that MC doesn't like. Beware of the current TWS version- 904.

janus
Posts: 755
Joined: 25 May 2009
Has thanked: 41 times
Been thanked: 84 times

Postby janus » 19 Apr 2010

Have you used the IB gateway recently for the first time? I did and wondering if it's something to do with that. You could rename the jts.ini file to force tws to create a new one to see if it helps.

User avatar
Bruce DeVault
Posts: 438
Joined: 19 Jan 2010
Location: Washington DC
Been thanked: 2 times
Contact:

Postby Bruce DeVault » 19 Apr 2010

If you suspect it may have something to do with a TWS setting that's changed, it's easy to specify a new settings directory e.g. C:\JTS2 instead of C:\JTS to force TWS to start completely anew (you can always switch back to your previous settings) - this will allow you to easily check if you think something's gone amiss because of trying the gateway. (I don't think that SHOULD be the case, but I am very interested to hear your experiences with this so please do let us know how it works out.)

janus
Posts: 755
Joined: 25 May 2009
Has thanked: 41 times
Been thanked: 84 times

Postby janus » 19 Apr 2010

So far I'm not having any problems since switching off "use server timestamps". If I do have problems again I will try your suggestion (although just renaming the jts.ini file has worked for me in the past with other problems - for example I tried the VPN option to play with but tws stopped working even when I went back to the normal connection method).

sptrader
Posts: 640
Joined: 09 Apr 2010
Location: Colorado
Has thanked: 403 times
Been thanked: 239 times
Contact:

Postby sptrader » 19 Apr 2010

If you suspect it may have something to do with a TWS setting that's changed, it's easy to specify a new settings directory e.g. C:\JTS2 instead of C:\JTS to force TWS to start completely anew (you can always switch back to your previous settings) - this will allow you to easily check if you think something's gone amiss because of trying the gateway. (I don't think that SHOULD be the case, but I am very interested to hear your experiences with this so please do let us know how it works out.)


** I DID try the gateway over the weekend(MC wouldn't connect to it, even after changing the port), I didn't think it could be related, now I wonder.

janus
Posts: 755
Joined: 25 May 2009
Has thanked: 41 times
Been thanked: 84 times

Postby janus » 19 Apr 2010

** I DID try the gateway over the weekend(MC wouldn't connect to it, even after changing the port), I didn't think it could be related, now I wonder.


Mine did work but as I posted before I prefer the full tws. Now I will rename the jts.ini and try again with "use server timestamps turned" on to see how it goes.

janus
Posts: 755
Joined: 25 May 2009
Has thanked: 41 times
Been thanked: 84 times

Postby janus » 19 Apr 2010

Using a fresh copy of tws working directory didn't help. Still have the problem with "use server timestamps" turned on. Now turned if off and all OK again. So, it appears attempting to use the IB gateway has nothing to do with it.

sptrader
Posts: 640
Joined: 09 Apr 2010
Location: Colorado
Has thanked: 403 times
Been thanked: 239 times
Contact:

Postby sptrader » 19 Apr 2010

Using a fresh copy of tws working directory didn't help. Still have the problem with "use server timestamps" turned on. Now turned if off and all OK again. So, it appears attempting to use the IB gateway has nothing to do with it.


** Good , thanks for the update, I'm still going back to the previous version of TWS though, it ran flawlessly for many weeks. (I'll probably rename the tws dir too)

cyberdad
Posts: 36
Joined: 10 Aug 2009

Postby cyberdad » 30 Apr 2010

I could well be off-topic here but since time-stamps are all about data consistency across multiple clients (or even between client-server), I ll say it anyway.

I noticed long ago that the feed from IB as it is registered by MC could have a lot of minor differences among different clients. Even clients connected to the same IB server. At first I thought it was an IB problem but then I found out it was much simpler.

The minor differences in OHLC data is a result of not synchronized PC time.
After installing a time synchronizing service everything fell into place. Now the data bars are identical among different clients and almost identical to IB charts as well.

I guess not many people run multiple clients but for those who do, I totally recommend installing a time-synchronizing service like this one

http://www.meinberg.de/english/info/ntp.htm

User avatar
Bruce DeVault
Posts: 438
Joined: 19 Jan 2010
Location: Washington DC
Been thanked: 2 times
Contact:

Postby Bruce DeVault » 30 Apr 2010

In real-time, IB does not supply timestamps with its "snapshot ticks" - as a consequence, the PC time is used in this case. This is a situation that's not true of most data feeds, it's an IB limitation.

It's actually documented to be this way in the MultiCharts help:

Interactive Brokers does not provide timestamps for quotes. Because of this limitation the program has to assign the system time to every quote. Such a workaround may bias the true timestamp of the quote by the time difference between its sending and receiving; in most cases the delay is fractions of a second


You're absolutely right, if you're using IB as a data source, you should make sure your time is synchronized and correct, to both minimize error relative to the time stamps at their end, and also maximize reproducibility on other machines.

You should also be aware that latency between IB and you affects the bars that are built, so if your PC is located far from IB on the internet (as measured in "hops" or milliseconds) you may have less reliability in the bars that are built than others have.


Return to “MultiCharts”