+1 888 340 6572 



Post new topic Reply to topic 
IB backfill problem (pacing violation)   [ ]
Author Message
#1 posted 27 Dec 2006 

Status: Offline
Hi,

I am using the latest MC beta with IB TWS 865.7 and I cannot get historical tick data for my charts.

Quote Manager says : TWS error message: Id=19; Code=162; Description=Historical Market Data Service error message: Historical data request pacing violation

I haven't seen this message before and whatever I tried (restarting etc) it keeps coming back. Any idea what this is and how to fix this? Does anyone else have similar problems? Is this a fault of IB or the new MC?

Thank you


#2 posted 28 Dec 2006 
User avatar

Status: Offline
Posts: 796
Thanks Received: 0 time
Thanks Given: 0 time
Joined: 08 Dec 2006
Alexi,

Our developers are analyzing this problem and I'll post here their conclusion as soon as they determine its reasons.


#3 posted 28 Dec 2006 

Status: Offline
Thanks Kate, I will watch this space for the suggested fix.


#4 posted 29 Dec 2006 
User avatar

Status: Offline
Posts: 796
Thanks Received: 0 time
Thanks Given: 0 time
Joined: 08 Dec 2006
This pacing violation is an IB innovation and it means that after a number of data requests IB stops processing the incoming requests and after some time starts again. So for now after some time IB begins to return data in MultiCharts.

We will take this into consideration and try to adjust to this IB innovation to minimize the waiting time after this limitation occurs.


#5 posted 29 Dec 2006 

Status: Offline
Posts: 87
Thanks Received: 0 time
Thanks Given: 0 time
Joined: 07 Dec 2006
Quote:
This pacing violation is an IB innovation and it means that after a number of data requests IB stops processing the incoming requests and after some time starts again. So for now after some time IB begins to return data in MultiCharts.

We will take this into consideration and try to adjust to this IB innovation to minimize the waiting time after this limitation occurs.


This appears to be an IB issue. I've been using the same charts/symbols for the last two weeks and now this morning I'm getting the pacing violation error and charts will not load. I've removed all but one workspace, disconnected all but one symbol, and restarted TWS without success. This issue is very frustrating. There is nothing more I can do and my charts are left "Waiting for data...."

~mojo


[12/29/2006 07:59:46] Establishing connection to TWS: Datafeed Version= 1.0.105.105, TWS Version= 8657, JTS Path= C:\Jts, JTS FileName= C:\Jts\jts.jar
[12/29/2006 07:59:46] Connecting to TWS... Host= 127.0.0.1, HostIP= 127.0.0.1, Port= 7496, ClientID= 10671
[12/29/2006 07:59:46] Connection is OK.
[12/29/2006 07:59:46] Connecting to real-time data for 6EH7: TransID= 1
[12/29/2006 07:59:46] TWS warning message: Id= -1; Code= 2104; Description= Market data farm connection is OK:usfuture
[12/29/2006 07:59:46] TWS warning message: Id= -1; Code= 2107; Description= HMDS data farm connection is inactive but should be available upon demand.:ushmds2a
[12/29/2006 07:59:49] Connecting to real-time data for ER2H7: TransID= 6
[12/29/2006 07:59:58] Data request for 6EH7 1 Day Trade [12/25/2006 13:59:58 - 12/29/2006 13:59:58] (Id= 2)
[12/29/2006 07:59:59] TWS warning message: Id= -1; Code= 2106; Description= HMDS data farm connection is OK:ushmds2a
[12/29/2006 07:59:59] TWS error message: Id= 2; Code= 165; Description= Historical Market Data Service query message:HMDS server connection was successful.
[12/29/2006 07:59:59] Data request for 6EH7 1 Tick Trade [12/29/2006 13:57:59 - 12/29/2006 13:59:59] (Id= 17)
[12/29/2006 07:59:59] Sending 6EH7 1 Tick Trade history to data server: 83 quotes downloaded (Key= 6EH7;T;TICK;1;GLOBEX;FUT;USD;): TransID= 4
[12/29/2006 07:59:59] Sending 6EH7 1 Tick Trade history to data server: 3 quotes downloaded (Key= 6EH7;T;TICK;1;GLOBEX;FUT;USD;): TransID= 5
[12/29/2006 07:59:59] Data request for 6EH7 1 Tick Trade [12/29/2006 13:53:00 - 12/29/2006 13:54:00] (Id= 4)
[12/29/2006 08:00:00] Sending 6EH7 1 Tick Trade history to data server: 2 quotes downloaded (Key= 6EH7;T;TICK;1;GLOBEX;FUT;USD;): TransID= 7
[12/29/2006 08:00:00] Data request for 6EH7 1 Tick Trade [12/29/2006 13:04:40 - 12/29/2006 13:38:00] (Id= 5)
[12/29/2006 08:00:01] Data request for 6EH7 1 Tick Trade [12/29/2006 12:31:20 - 12/29/2006 13:04:40] (Id= 20)
[12/29/2006 08:00:02] Data request for 6EH7 1 Tick Trade [12/29/2006 11:58:00 - 12/29/2006 12:31:20] (Id= 21)
[12/29/2006 08:00:03] Data request for 6EH7 1 Tick Trade [12/29/2006 11:24:40 - 12/29/2006 11:58:00] (Id= 22)
[12/29/2006 08:00:03] TWS error message: Id= 22; Code= 162; Description= Historical Market Data Service error message:Historical data request pacing violation


#6 posted 29 Dec 2006 

Status: Offline
Posts: 87
Thanks Received: 0 time
Thanks Given: 0 time
Joined: 07 Dec 2006
As a follow-up, Ensign Software's IB refresh also failed. I had to update using their DTN feed.

This is definitely an IB problem and I don't think there is anything MC can do to resolve this, other than display charts with available data.


~mojo


#7 posted 29 Dec 2006 

Status: Offline
I am facing this problem too...and my charts are taking a very very long time to load, due to this


#8 posted 29 Dec 2006 

Status: Offline
Posts: 89
Thanks Received: 0 time
Thanks Given: 0 time
Joined: 05 Nov 2006
Quote:
As a follow-up, Ensign Software's IB refresh also failed. I had to update using their DTN feed.

This is definitely an IB problem and I don't think there is anything MC can do to resolve this, other than display charts with available data.


~mojo


SC has said that IB's backfill has ALWAYS been unreliable. It's not the charting program

www.SC.com/index.php?file=Emai ... hp&aid=def

They recommended using DTN, OpenTick, TransAct, etc to backfill historical tick data.

_________________
Josh

MC beta version multicharts.5.0.2258.206_beta6.exe

macbook w/ VMware Fusion 2.00; WinxP sp3, 2 gig RAM, core 2 duo (using one core). Maybe one day will dual-boot, but for now emulation works excellent.


#9 posted 29 Dec 2006 

Status: Offline
@ Kate

Is there an way to include DTN Market Access into Multicharts like linnsoft with the same cost?

www.linnsoft.com/dtnma/index.htm


#10 posted 30 Dec 2006 

Status: Offline
Thanks for the reply Kate. It really is totally unacceptable from IB the way they are changing and 'upgrading' their software. Everything seemed to run well until their last upgrade...

Having read the various posts in other threads about the slow speed of their backfill I thought that I would just be a bit more patient and wait for the charts to fill. But with the new issue identified here makes backfilling almost impossible and totally messes up MC charts (especially if one is using several symbols in multiple workspaces as I do, where historical gap filling is no longer practical as the wait can be infinite!).

It would be great if you guys could somehow circumvent this of course. In the meantime I wrote them an angry complaint letter + email regarding primarily this issue. I wonder if they will listen though, they are famous for their nonexistent customer support. I would suggest that others that are also affected do the same and complain to them directly in hopes that this change to the way their historical servers work can be reversed.


Thanks again
Alexi


#11 posted 30 Dec 2006 

Status: Offline
Thanks for the reply Kate. It really is totally unacceptable from IB the way they are changing and 'upgrading' their software. Everything seemed to run well until their last upgrade...

Having read the various posts in other threads about the slow speed of their backfill I thought that I would just be a bit more patient and wait for the charts to fill. But with the new issue identified here makes backfilling almost impossible and totally messes up MC charts (especially if one is using several symbols in multiple workspaces as I do, where historical gap filling is no longer practical as the wait can be infinite!).

It would be great if you guys could somehow circumvent this of course. In the meantime I wrote them an angry complaint letter + email regarding primarily this issue. I wonder if they will listen though, they are famous for their nonexistent customer support. I would suggest that others that are also affected do the same and complain to them directly in hopes that this change to the way their historical servers work can be reversed.


Thanks again
Alexi


#12 posted 30 Dec 2006 

Status: Offline
Posts: 74
Thanks Received: 0 time
Thanks Given: 0 time
Joined: 24 Jul 2005
Here is a quote from another IB 3rd party list in which the poster relays this is direct from IB -

*****

"We allow 60 backfill requests within a 5 minute
period. In the past, once you exceeded that limit we queued up
additional request but delayed data to maintain a ratio of 60 per 5
minute period. Keeping the additional request in a queue and delayed,
caused stress on our system. Request that exceed the limit will now be
rejected."

*****


Kate wrote:

Quote:
This pacing violation is an IB innovation and it means that after a number of data requests IB stops processing the incoming requests and after some time starts again. So for now after some time IB begins to return data in MultiCharts.

We will take this into consideration and try to adjust to this IB innovation to minimize the waiting time after this limitation occurs.


#13 posted 30 Dec 2006 

Status: Offline
Posts: 9
Thanks Received: 0 time
Thanks Given: 0 time
Joined: 23 Apr 2006
I would like a data feed suggestion from a MC user that is using a reliable data feed provider. It seems that I notice many complaints from IB users. These complaints appear not only here but other forums. I am currently using IB but would like an opinion from someone using a provider with which they are satisfied. Thanks and Happy New Year to everyone.


#14 posted 03 Jan 2007 

Status: Offline
Posts: 488
Thanks Received: 23 times
Thanks Given: 4 times
Joined: 04 Aug 2006
Hi do you plan to release a patch soon to deal with this? Or will the current version keep trying and finally get data again after the 5 minutes ha passed? (I still havent upgraded to the latest beta).

On the bright side IB's backfill does seem a lot faster than it used to be now they dont queue requests.

Cheers,
Nick.


#15 posted 03 Jan 2007 

Status: Offline
Posts: 366
Thanks Received: 12 times
Thanks Given: 7 times
Joined: 23 Nov 2005
I'm on the previous TWS and am not seeing this problem. Have any of you tried rolling back to the previous TWS?


#16 posted 03 Jan 2007 
User avatar

Status: Offline
Posts: 5182
Thanks Received: 1424 times
Thanks Given: 856 times
Joined: 29 Aug 2006
Quote:
I would like a data feed suggestion from a MC user that is using a reliable data feed provider. It seems that I notice many complaints from IB users. These complaints appear not only here but other forums. I am currently using IB but would like an opinion from someone using a provider with which they are satisfied. Thanks and Happy New Year to everyone.

I was a long time subscriber to eSignal. I can tell you: Nobody is perfect, they also have hang ups, data gaps, drop outs...


#17 posted 04 Jan 2007 

Status: Offline
Posts: 488
Thanks Received: 23 times
Thanks Given: 4 times
Joined: 04 Aug 2006
I agree TJ though this issue is something that is consistant. Basically IB are enforcing the API in a more rigerous way now.Actually the problem really is they changed things without much warning.

The thing I like about IB is that in fast markets they don't lag nearly as badly as esignal (because they aggregate data). often I would see quotes from IB a good 10 seconds before they appeared on Esignal. Oh, that and the fact that e$ignal costs close to $350/mo if you subscribe to a couple of 'foreign' exchanges and forex.

Things would have to get pretty bad for me to go back to Esignal. Does DTN's IQfeed offer european exchanges yet ?


#18 posted 04 Jan 2007 
User avatar

Status: Offline
Posts: 1946
Thanks Received: 31 times
Thanks Given: 2 times
Joined: 28 Jul 2005
Dear Nick,

Quote:
Does DTN's IQfeed offer european exchanges yet ?


Please check these links for IQFeed services and pricing information:

http://www.iqfeed.net/symbolguide/index ... on=support
http://www.iqfeed.net/index.cfm?display ... ction=fees

There are some european instruments available.


#19 posted 05 Jan 2007 
User avatar

Status: Offline
Posts: 796
Thanks Received: 0 time
Thanks Given: 0 time
Joined: 08 Dec 2006
Quote:

Is there an way to include DTN Market Access into Multicharts like linnsoft with the same cost?

www.linnsoft.com/dtnma/index.htm


MultiCharts doesn't support DTN Market Access $12 service. Please use services listed at:
http://www.iqfeed.net/index.cfm?display ... ction=fees


#20 posted 05 Jan 2007 
User avatar

Status: Offline
Posts: 9
Thanks Received: 0 time
Thanks Given: 0 time
Joined: 05 Jan 2007
Just to add to this, the DTN Market Access service on Linnsoft is actually a wholesale service that Linnsoft re-sells. The $12 per month is their pricing and it isn't a service available with an API like MC uses or directly from DTN.

The best bet for active traders who rely on data is to always utilize 2 services. One of the nice things for us is we have a few 3rd party developers who have created a monitoring tool that compares our IQFeed to that of various other feeds on a continual basis (tick by tick). This has been a great sanity check for us, and keeps us on our toes to ensure we continue to have a fast and reliable data. Plus, we have dedicated staff working on our historical data to keep it clean. IB is good for the money, but if you rely on data to trade, I can't think of anyone who wouldn't recommend comitting a few good dollars a month to ensure you have a backup and/or mor reliable option when you need it.

DTN does have some international exchanges, but all on the futures markets (Eurex, Simex, Euronext, LME, etc). At this time we don't plan to add any non-north american stock exchanges however.

_________________
Jay Froscheiser
DTN
www.iqfeed.net/tssupport
Sales: 800-475-4755
Sales Chat: www.iqfeed.net

Customer/Technical Support for DTN customers is available via 800#, Email, RT Chat and Forums.


#21 posted 08 Jan 2007 

Status: Offline
Quote:
Here is a quote from another IB 3rd party list in which the poster relays this is direct from IB -

*****

"We allow 60 backfill requests within a 5 minute
period. In the past, once you exceeded that limit we queued up
additional request but delayed data to maintain a ratio of 60 per 5
minute period. Keeping the additional request in a queue and delayed,
caused stress on our system. Request that exceed the limit will now be
rejected."

*****


Kate wrote:

Quote:
This pacing violation is an IB innovation and it means that after a number of data requests IB stops processing the incoming requests and after some time starts again. So for now after some time IB begins to return data in MultiCharts.

We will take this into consideration and try to adjust to this IB innovation to minimize the waiting time after this limitation occurs.


IB is only refusing excessive backfill request. It shouldn't affect realtime quotes. Apparently IB just freezes and stops realtime charting as well.



Post new topic Reply to topic