IB backfill problem (pacing violation)  [SOLVED]

Questions about MultiCharts and user contributed studies.
Alexi

IB backfill problem (pacing violation)

Postby Alexi » 27 Dec 2006

Hi,

I am using the latest MC beta with IB TWS 865.7 and I cannot get historical tick data for my charts.

Quote Manager says : TWS error message: Id=19; Code=162; Description=Historical Market Data Service error message: Historical data request pacing violation

I haven't seen this message before and whatever I tried (restarting etc) it keeps coming back. Any idea what this is and how to fix this? Does anyone else have similar problems? Is this a fault of IB or the new MC?

Thank you

User avatar
Kate
Posts: 758
Joined: 08 Dec 2006

Postby Kate » 28 Dec 2006

Alexi,

Our developers are analyzing this problem and I'll post here their conclusion as soon as they determine its reasons.

Alexi

Postby Alexi » 28 Dec 2006

Thanks Kate, I will watch this space for the suggested fix.

User avatar
Kate
Posts: 758
Joined: 08 Dec 2006

Postby Kate » 29 Dec 2006

This pacing violation is an IB innovation and it means that after a number of data requests IB stops processing the incoming requests and after some time starts again. So for now after some time IB begins to return data in MultiCharts.

We will take this into consideration and try to adjust to this IB innovation to minimize the waiting time after this limitation occurs.

mojotrader
Posts: 87
Joined: 07 Dec 2006

Postby mojotrader » 29 Dec 2006

This pacing violation is an IB innovation and it means that after a number of data requests IB stops processing the incoming requests and after some time starts again. So for now after some time IB begins to return data in MultiCharts.

We will take this into consideration and try to adjust to this IB innovation to minimize the waiting time after this limitation occurs.
This appears to be an IB issue. I've been using the same charts/symbols for the last two weeks and now this morning I'm getting the pacing violation error and charts will not load. I've removed all but one workspace, disconnected all but one symbol, and restarted TWS without success. This issue is very frustrating. There is nothing more I can do and my charts are left "Waiting for data...."

~mojo


[12/29/2006 07:59:46] Establishing connection to TWS: Datafeed Version= 1.0.105.105, TWS Version= 8657, JTS Path= C:\Jts, JTS FileName= C:\Jts\jts.jar
[12/29/2006 07:59:46] Connecting to TWS... Host= 127.0.0.1, HostIP= 127.0.0.1, Port= 7496, ClientID= 10671
[12/29/2006 07:59:46] Connection is OK.
[12/29/2006 07:59:46] Connecting to real-time data for 6EH7: TransID= 1
[12/29/2006 07:59:46] TWS warning message: Id= -1; Code= 2104; Description= Market data farm connection is OK:usfuture
[12/29/2006 07:59:46] TWS warning message: Id= -1; Code= 2107; Description= HMDS data farm connection is inactive but should be available upon demand.:ushmds2a
[12/29/2006 07:59:49] Connecting to real-time data for ER2H7: TransID= 6
[12/29/2006 07:59:58] Data request for 6EH7 1 Day Trade [12/25/2006 13:59:58 - 12/29/2006 13:59:58] (Id= 2)
[12/29/2006 07:59:59] TWS warning message: Id= -1; Code= 2106; Description= HMDS data farm connection is OK:ushmds2a
[12/29/2006 07:59:59] TWS error message: Id= 2; Code= 165; Description= Historical Market Data Service query message:HMDS server connection was successful.
[12/29/2006 07:59:59] Data request for 6EH7 1 Tick Trade [12/29/2006 13:57:59 - 12/29/2006 13:59:59] (Id= 17)
[12/29/2006 07:59:59] Sending 6EH7 1 Tick Trade history to data server: 83 quotes downloaded (Key= 6EH7;T;TICK;1;GLOBEX;FUT;USD;): TransID= 4
[12/29/2006 07:59:59] Sending 6EH7 1 Tick Trade history to data server: 3 quotes downloaded (Key= 6EH7;T;TICK;1;GLOBEX;FUT;USD;): TransID= 5
[12/29/2006 07:59:59] Data request for 6EH7 1 Tick Trade [12/29/2006 13:53:00 - 12/29/2006 13:54:00] (Id= 4)
[12/29/2006 08:00:00] Sending 6EH7 1 Tick Trade history to data server: 2 quotes downloaded (Key= 6EH7;T;TICK;1;GLOBEX;FUT;USD;): TransID= 7
[12/29/2006 08:00:00] Data request for 6EH7 1 Tick Trade [12/29/2006 13:04:40 - 12/29/2006 13:38:00] (Id= 5)
[12/29/2006 08:00:01] Data request for 6EH7 1 Tick Trade [12/29/2006 12:31:20 - 12/29/2006 13:04:40] (Id= 20)
[12/29/2006 08:00:02] Data request for 6EH7 1 Tick Trade [12/29/2006 11:58:00 - 12/29/2006 12:31:20] (Id= 21)
[12/29/2006 08:00:03] Data request for 6EH7 1 Tick Trade [12/29/2006 11:24:40 - 12/29/2006 11:58:00] (Id= 22)
[12/29/2006 08:00:03] TWS error message: Id= 22; Code= 162; Description= Historical Market Data Service error message:Historical data request pacing violation

mojotrader
Posts: 87
Joined: 07 Dec 2006

  [SOLVED]

Postby mojotrader » 29 Dec 2006

As a follow-up, Ensign Software's IB refresh also failed. I had to update using their DTN feed.

This is definitely an IB problem and I don't think there is anything MC can do to resolve this, other than display charts with available data.


~mojo

gerimegaly

Postby gerimegaly » 29 Dec 2006

I am facing this problem too...and my charts are taking a very very long time to load, due to this

ybfjax
Posts: 89
Joined: 05 Nov 2006
Contact:

it's a IB problem

Postby ybfjax » 29 Dec 2006

As a follow-up, Ensign Software's IB refresh also failed. I had to update using their DTN feed.

This is definitely an IB problem and I don't think there is anything MC can do to resolve this, other than display charts with available data.


~mojo
SC has said that IB's backfill has ALWAYS been unreliable. It's not the charting program

http://www.SC.com/index.php?fi ... hp&aid=def

They recommended using DTN, OpenTick, TransAct, etc to backfill historical tick data.

Guest

Postby Guest » 29 Dec 2006

@ Kate

Is there an way to include DTN Market Access into Multicharts like linnsoft with the same cost?

www.linnsoft.com/dtnma/index.htm

Alexi

Postby Alexi » 30 Dec 2006

Thanks for the reply Kate. It really is totally unacceptable from IB the way they are changing and 'upgrading' their software. Everything seemed to run well until their last upgrade...

Having read the various posts in other threads about the slow speed of their backfill I thought that I would just be a bit more patient and wait for the charts to fill. But with the new issue identified here makes backfilling almost impossible and totally messes up MC charts (especially if one is using several symbols in multiple workspaces as I do, where historical gap filling is no longer practical as the wait can be infinite!).

It would be great if you guys could somehow circumvent this of course. In the meantime I wrote them an angry complaint letter + email regarding primarily this issue. I wonder if they will listen though, they are famous for their nonexistent customer support. I would suggest that others that are also affected do the same and complain to them directly in hopes that this change to the way their historical servers work can be reversed.


Thanks again
Alexi

Alexi

Postby Alexi » 30 Dec 2006

Thanks for the reply Kate. It really is totally unacceptable from IB the way they are changing and 'upgrading' their software. Everything seemed to run well until their last upgrade...

Having read the various posts in other threads about the slow speed of their backfill I thought that I would just be a bit more patient and wait for the charts to fill. But with the new issue identified here makes backfilling almost impossible and totally messes up MC charts (especially if one is using several symbols in multiple workspaces as I do, where historical gap filling is no longer practical as the wait can be infinite!).

It would be great if you guys could somehow circumvent this of course. In the meantime I wrote them an angry complaint letter + email regarding primarily this issue. I wonder if they will listen though, they are famous for their nonexistent customer support. I would suggest that others that are also affected do the same and complain to them directly in hopes that this change to the way their historical servers work can be reversed.


Thanks again
Alexi

traderstuff
Posts: 68
Joined: 24 Jul 2005

Postby traderstuff » 30 Dec 2006

Here is a quote from another IB 3rd party list in which the poster relays this is direct from IB -

*****

"We allow 60 backfill requests within a 5 minute
period. In the past, once you exceeded that limit we queued up
additional request but delayed data to maintain a ratio of 60 per 5
minute period. Keeping the additional request in a queue and delayed,
caused stress on our system. Request that exceed the limit will now be
rejected."

*****


Kate wrote:
This pacing violation is an IB innovation and it means that after a number of data requests IB stops processing the incoming requests and after some time starts again. So for now after some time IB begins to return data in MultiCharts.

We will take this into consideration and try to adjust to this IB innovation to minimize the waiting time after this limitation occurs.

mm
Posts: 9
Joined: 23 Apr 2006
Location: Texas

Postby mm » 30 Dec 2006

I would like a data feed suggestion from a MC user that is using a reliable data feed provider. It seems that I notice many complaints from IB users. These complaints appear not only here but other forums. I am currently using IB but would like an opinion from someone using a provider with which they are satisfied. Thanks and Happy New Year to everyone.

Nick
Posts: 496
Joined: 04 Aug 2006
Has thanked: 4 times
Been thanked: 24 times

Postby Nick » 03 Jan 2007

Hi do you plan to release a patch soon to deal with this? Or will the current version keep trying and finally get data again after the 5 minutes ha passed? (I still havent upgraded to the latest beta).

On the bright side IB's backfill does seem a lot faster than it used to be now they dont queue requests.

Cheers,
Nick.

momentum
Posts: 324
Joined: 23 Nov 2005
Has thanked: 9 times
Been thanked: 14 times

Postby momentum » 03 Jan 2007

I'm on the previous TWS and am not seeing this problem. Have any of you tried rolling back to the previous TWS?

User avatar
TJ
Posts: 7740
Joined: 29 Aug 2006
Location: Global Citizen
Has thanked: 1033 times
Been thanked: 2221 times

Postby TJ » 03 Jan 2007

I would like a data feed suggestion from a MC user that is using a reliable data feed provider. It seems that I notice many complaints from IB users. These complaints appear not only here but other forums. I am currently using IB but would like an opinion from someone using a provider with which they are satisfied. Thanks and Happy New Year to everyone.
I was a long time subscriber to eSignal. I can tell you: Nobody is perfect, they also have hang ups, data gaps, drop outs...

Nick
Posts: 496
Joined: 04 Aug 2006
Has thanked: 4 times
Been thanked: 24 times

Postby Nick » 04 Jan 2007

I agree TJ though this issue is something that is consistant. Basically IB are enforcing the API in a more rigerous way now.Actually the problem really is they changed things without much warning.

The thing I like about IB is that in fast markets they don't lag nearly as badly as esignal (because they aggregate data). often I would see quotes from IB a good 10 seconds before they appeared on Esignal. Oh, that and the fact that e$ignal costs close to $350/mo if you subscribe to a couple of 'foreign' exchanges and forex.

Things would have to get pretty bad for me to go back to Esignal. Does DTN's IQfeed offer european exchanges yet ?

User avatar
Andrew Kirillov
Posts: 1589
Joined: 28 Jul 2005
Has thanked: 2 times
Been thanked: 31 times
Contact:

Postby Andrew Kirillov » 04 Jan 2007

Dear Nick,
Does DTN's IQfeed offer european exchanges yet ?
Please check these links for IQFeed services and pricing information:

http://www.iqfeed.net/symbolguide/index ... on=support
http://www.iqfeed.net/index.cfm?display ... ction=fees

There are some european instruments available.

User avatar
Kate
Posts: 758
Joined: 08 Dec 2006

Postby Kate » 05 Jan 2007


Is there an way to include DTN Market Access into Multicharts like linnsoft with the same cost?

www.linnsoft.com/dtnma/index.htm
MultiCharts doesn't support DTN Market Access $12 service. Please use services listed at:
http://www.iqfeed.net/index.cfm?display ... ction=fees

DTN IQFeed
Posts: 9
Joined: 05 Jan 2007
Location: Omaha, NE
Contact:

Postby DTN IQFeed » 05 Jan 2007

Just to add to this, the DTN Market Access service on Linnsoft is actually a wholesale service that Linnsoft re-sells. The $12 per month is their pricing and it isn't a service available with an API like MC uses or directly from DTN.

The best bet for active traders who rely on data is to always utilize 2 services. One of the nice things for us is we have a few 3rd party developers who have created a monitoring tool that compares our IQFeed to that of various other feeds on a continual basis (tick by tick). This has been a great sanity check for us, and keeps us on our toes to ensure we continue to have a fast and reliable data. Plus, we have dedicated staff working on our historical data to keep it clean. IB is good for the money, but if you rely on data to trade, I can't think of anyone who wouldn't recommend comitting a few good dollars a month to ensure you have a backup and/or mor reliable option when you need it.

DTN does have some international exchanges, but all on the futures markets (Eurex, Simex, Euronext, LME, etc). At this time we don't plan to add any non-north american stock exchanges however.

Guest

Postby Guest » 08 Jan 2007

Here is a quote from another IB 3rd party list in which the poster relays this is direct from IB -

*****

"We allow 60 backfill requests within a 5 minute
period. In the past, once you exceeded that limit we queued up
additional request but delayed data to maintain a ratio of 60 per 5
minute period. Keeping the additional request in a queue and delayed,
caused stress on our system. Request that exceed the limit will now be
rejected."

*****


Kate wrote:
This pacing violation is an IB innovation and it means that after a number of data requests IB stops processing the incoming requests and after some time starts again. So for now after some time IB begins to return data in MultiCharts.

We will take this into consideration and try to adjust to this IB innovation to minimize the waiting time after this limitation occurs.
IB is only refusing excessive backfill request. It shouldn't affect realtime quotes. Apparently IB just freezes and stops realtime charting as well.


Return to “MultiCharts”