Custom Futures Tick chart causes IB pacing violation  [SOLVED]

Questions about MultiCharts and user contributed studies.
wzero
Posts: 142
Joined: 10 Jul 2014
Has thanked: 12 times
Been thanked: 10 times

Custom Futures Tick chart causes IB pacing violation

Postby wzero » 08 Oct 2014

I use MC 9. I can't make Custom (continuous) Futures works with Tick/Second charts. It will 100% cause IB "pacing violation", the tick or second charts would never load. I know IB data feed has limitation but all other normal (non-custom) instruments worked very well, and Custom Futures with 1 min above period charts worked well too. I wonder why only Custom Futures combine Tick/second charts cause problem. I checked the QuoteManager log, it shows Custom Futures Tick charts will request Tick data about 10 days ago. This seems unreasonable to me, because my tick chart is only intraday (500 bars).

Nick
Posts: 490
Joined: 04 Aug 2006
Has thanked: 4 times
Been thanked: 24 times

Re: Custom Futures Tick chart causes IB pacing violation

Postby Nick » 08 Oct 2014

I think it is because MC requests data from previous (expired) contracts (future ones too). See http://www.multicharts.com/discussion/viewtopic.php?f=1&t=47315#p109018 why this is requested if it is not required for back adjustment I am not sure.

wzero
Posts: 142
Joined: 10 Jul 2014
Has thanked: 12 times
Been thanked: 10 times

Re: Custom Futures Tick chart causes IB pacing violation

Postby wzero » 08 Oct 2014

Because 1min and above periods have no problem, only tick/second charts have persistent pacing violation. I think even for a tick chart it should not look pervious TICK data for merging reason if this is the problem.

hilbert
Posts: 222
Joined: 17 Aug 2011
Has thanked: 76 times
Been thanked: 64 times

Re: Custom Futures Tick chart causes IB pacing violation

Postby hilbert » 08 Oct 2014

wzero wrote:I use MC 9. I can't make Custom (continuous) Futures works with Tick/Second charts. It will 100% cause IB "pacing violation", the tick or second charts would never load. I know IB data feed has limitation but all other normal (non-custom) instruments worked very well, and Custom Futures with 1 min above period charts worked well too. I wonder why only Custom Futures combine Tick/second charts cause problem. I checked the QuoteManager log, it shows Custom Futures Tick charts will request Tick data about 10 days ago. This seems unreasonable to me, because my tick chart is only intraday (500 bars).

can you post some screenshots please, both showing pacing violation for custom contracts and not causing pacing violation for normal (non-custom) instruments. It will be helpful to investigate then. Thanks.

wzero
Posts: 142
Joined: 10 Jul 2014
Has thanked: 12 times
Been thanked: 10 times

Re: Custom Futures Tick chart causes IB pacing violation

Postby wzero » 08 Oct 2014

Screenshot and log attached.
The log file shows it request a lot of tick data request for 9/24/2014, 9/25/2015 until pacing violation. But today is 8/10/2014, that data is not displayed in this chart.
Attachments
attach.zip
(53.45 KiB) Downloaded 142 times

napalm
Posts: 12
Joined: 15 Aug 2006
Location: London
Contact:

Re: Custom Futures Tick chart causes IB pacing violation

Postby napalm » 09 Oct 2014

I have same problem . perhaps the option to turn the message off would be agood idea. Its very annoying.

hilbert
Posts: 222
Joined: 17 Aug 2011
Has thanked: 76 times
Been thanked: 64 times

Re: Custom Futures Tick chart causes IB pacing violation

Postby hilbert » 09 Oct 2014

wzero wrote:Screenshot and log attached.
The log file shows it request a lot of tick data request for 9/24/2014, 9/25/2015 until pacing violation. But today is 8/10/2014, that data is not displayed in this chart.


I think its a bug and MC team should release a fix. When plotting minute data, this is just an inconvenience however if you can't plot tick data at all, then it means its a critical bug for you and MC team will fix this.

User avatar
Andrew MultiCharts
Posts: 1587
Joined: 11 Oct 2011
Has thanked: 931 times
Been thanked: 557 times

Re: Custom Futures Tick chart causes IB pacing violation

Postby Andrew MultiCharts » 09 Oct 2014

Hello,

Could you send us the following files to support@multicharts.com?
  1. The workspace to reproduce the situation.
  2. Export of used symbols (with data) from QuoteManager in .qmd archive: https://www.multicharts.com/trading-soft ... rting_Data
  3. The screenshot demonstrating the issue.

wzero
Posts: 142
Joined: 10 Jul 2014
Has thanked: 12 times
Been thanked: 10 times

Re: Custom Futures Tick chart causes IB pacing violation

Postby wzero » 09 Oct 2014

Hi Andrew, test logs sent by email. please check!

wzero
Posts: 142
Joined: 10 Jul 2014
Has thanked: 12 times
Been thanked: 10 times

Re: Custom Futures Tick chart causes IB pacing violation

Postby wzero » 13 Oct 2014

I tested custom futures on CQG demo data feed, tick chart loads but need much longer time to fetch data than "normal" instruments.
On IB data feed, all minute above periods have no problem. Only tick, second charts endless pacing violation.

User avatar
Andrew MultiCharts
Posts: 1587
Joined: 11 Oct 2011
Has thanked: 931 times
Been thanked: 557 times

Re: Custom Futures Tick chart causes IB pacing violation  [SOLVED]

Postby Andrew MultiCharts » 14 Oct 2014

Indeed there can be some problems with pacing violation if you use "Bars Back/Days Back" option for defining Data Range of the chart. If you switch the option to "From ... To ...", there will be less tick data requests, though according the mechanism of building custom futures charts, the number of data requests is higher than from a chart with individual futures contracs

hilbert
Posts: 222
Joined: 17 Aug 2011
Has thanked: 76 times
Been thanked: 64 times

Re: Custom Futures Tick chart causes IB pacing violation

Postby hilbert » 14 Oct 2014

Andrew MultiCharts wrote:Indeed there can be some problems with pacing violation if you use "Bars Back/Days Back" option for defining Data Range of the chart. If you switch the option to "From ... To ...", there will be less tick data requests, though according the mechanism of building custom futures charts, the number of data requests is higher than from a chart with individual futures contracs

Are you saying following?

Case I: I am requesting tick data for 2 days back
Case II: I am requesting tick data from 12th October to 14th October.

Why should case I cause pacing violation and Case II not cause pacing violation (or will cause much less pacing violation)? Does it mean there is some bug in Case I?

User avatar
Andrew MultiCharts
Posts: 1587
Joined: 11 Oct 2011
Has thanked: 931 times
Been thanked: 557 times

Re: Custom Futures Tick chart causes IB pacing violation

Postby Andrew MultiCharts » 14 Oct 2014

hilbert wrote:Why should case I cause pacing violation and Case II not cause pacing violation (or will cause much less pacing violation)? Does it mean there is some bug in Case I?

Because of different ways how data is requested on chart.

hilbert
Posts: 222
Joined: 17 Aug 2011
Has thanked: 76 times
Been thanked: 64 times

Re: Custom Futures Tick chart causes IB pacing violation

Postby hilbert » 14 Oct 2014

Andrew MultiCharts wrote:Because of different ways how data is requested on chart.

Alright, thanks. So, the best practice is to use From...To method for selecting the dates.

I guess you guys should update the Wiki page recommending that From...To method is superior compared to days back method. It will be helpful for users and might lead to lesser queries in future. Thanks a lot!
These users thanked the author hilbert for the post:
Andrew MultiCharts

wzero
Posts: 142
Joined: 10 Jul 2014
Has thanked: 12 times
Been thanked: 10 times

Re: Custom Futures Tick chart causes IB pacing violation

Postby wzero » 14 Oct 2014

Hi, Andrew. I changed to tick chart data range from "800 bars back" to "today date - today date". The custom futures load immediately! So this solved the problem. Thanks for that!
But the minimal setting is one day's range, it loads all one days ticks chart data. Actually much more bars than before! Weird thing is it's almost no pacing violation now...
Andrew, do you suggest to use "date - date" instead of "N bars back" for other minutes/hours/daily charts too? Would that speed up loading?

User avatar
Andrew MultiCharts
Posts: 1587
Joined: 11 Oct 2011
Has thanked: 931 times
Been thanked: 557 times

Re: Custom Futures Tick chart causes IB pacing violation

Postby Andrew MultiCharts » 14 Oct 2014

wzero wrote:Andrew, do you suggest to use "date - date" instead of "N bars back" for other minutes/hours/daily charts too? Would that speed up loading?

For any chart using custom futures from IB.

seneca
Posts: 87
Joined: 02 Apr 2012
Has thanked: 36 times
Been thanked: 22 times

Re: Custom Futures Tick chart causes IB pacing violation

Postby seneca » 16 Oct 2014

Andrew MultiCharts wrote:Indeed there can be some problems with pacing violation if you use "Bars Back/Days Back" option for defining Data Range of the chart. If you switch the option to "From ... To ...", there will be less tick data requests, though according the mechanism of building custom futures charts, the number of data requests is higher than from a chart with individual futures contracs


I have changed my custom futures to "From.....To...." mode in order to avoid IB pacing violations. I use today's date as "From date", but in contrast to the "To date", we cannot make the "From date" date sticky with today's date. So you would have to continuously adjust the "From date", in order to avoid that the data range grows every day. So I want to suggest to implement an additional sticky button for the "From date".

User avatar
Andrew MultiCharts
Posts: 1587
Joined: 11 Oct 2011
Has thanked: 931 times
Been thanked: 557 times

Re: Custom Futures Tick chart causes IB pacing violation

Postby Andrew MultiCharts » 16 Oct 2014

seneca wrote:I have changed my custom futures to "From.....To...." mode in order to avoid IB pacing violations. I use today's date as "From date", but in contrast to the "To date", we cannot make the "From date" date sticky with today's date. So you would have to continuously adjust the "From date", in order to avoid that the data range grows every day. So I want to suggest to implement an additional sticky button for the "From date".

Please leave us such feature request.

arjfca
Posts: 1273
Joined: 23 Nov 2010
Has thanked: 721 times
Been thanked: 222 times

Re: Custom Futures Tick chart causes IB pacing violation

Postby arjfca » 16 Oct 2014

Have the same problem with Tick chart

Here is what I think happen

I installed MC 64 and I lost all data ( did not back them up)
MC had to rebuilt from scratch all the historic

Before, when I was looking at a lower time frame, MC was fetching in is own data and only request the missing bars, Now, MC need to go gather all the data

IB does not recommend to load tick chart for more than 1 day.

So put that need to fetch all data for the 8 charts that I normally open, may definitively cause a pacing violation.

Probably my own fault because I did not do the data backup

Martin

wzero
Posts: 142
Joined: 10 Jul 2014
Has thanked: 12 times
Been thanked: 10 times

Re: Custom Futures Tick chart causes IB pacing violation

Postby wzero » 16 Oct 2014

Instead of update the “from" date. I hope the next release can fix the "Bars back" and "Days back" issue.
Another issue of the "Bars back" is my 4 Hours candlestick chart can only display at maximum of about 200 bars for US Stocks. Even I increase Bars back to 800, only about 200 bars are displayed. Forex and Futures don't have this problem, only US stocks. Even for US stocks, if I use "From Date - To Date", no problem.

arjfca
Posts: 1273
Joined: 23 Nov 2010
Has thanked: 721 times
Been thanked: 222 times

Re: Custom Futures Tick chart causes IB pacing violation

Postby arjfca » 16 Oct 2014

wzero wrote:Instead of update the “from" date. I hope the next release can fix the "Bars back" and "Days back" issue.
Another issue of the "Bars back" is my 4 Hours candlestick chart can only display at maximum of about 200 bars for US Stocks. Even I increase Bars back to 800, only about 200 bars are displayed. Forex and Futures don't have this problem, only US stocks. Even for US stocks, if I use "From Date - To Date", no problem.

I think the problem is caused by IB limitation

Try to load 1 chart at the time with fewer bars. If you got pacing violation, turn off and on TWS. Load 200 bars then 400 then 600 etc

MC Should have the first 200 should then will request the next 200 to create the 400.

Or, load up all your chart and go do something else for a couple of hours. Historic should been rebuilt by the,

Can't say if it will resolve the problem. Yesterday, I was always stopped by the pacing violation, today did not happen yet.

Martin

wzero
Posts: 142
Joined: 10 Jul 2014
Has thanked: 12 times
Been thanked: 10 times

Re: Custom Futures Tick chart causes IB pacing violation

Postby wzero » 16 Oct 2014

The US Stocks 4 hours bar problem is not related to pacing violation. It just don't load more than about 200 bars data. It's not error on loading, it's no loading no display at all.

arjfca
Posts: 1273
Joined: 23 Nov 2010
Has thanked: 721 times
Been thanked: 222 times

Re: Custom Futures Tick chart causes IB pacing violation

Postby arjfca » 16 Oct 2014

wzero wrote:The US Stocks 4 hours bar problem is not related to pacing violation. It just don't load more than about 200 bars data. It's not error on loading, it's no loading no display at all.


Try to close all charts
Go in QuoteManager
Delete the cache and the instrument
Relaod the instrument
reconnect the instrument

Reload your chart

Go fetch historic in small chunk to avoid pacing violation.

I did write a request to ask MC to create internal rules to prevent pacing violation.

You could vote
https://www.multicharts.com/pm/viewissue ... no=MC-1770

Martin

wzero
Posts: 142
Joined: 10 Jul 2014
Has thanked: 12 times
Been thanked: 10 times

Re: Custom Futures Tick chart causes IB pacing violation

Postby wzero » 16 Oct 2014

The US Stocks 4 hours bars problem is not related to the Cache. It seems hourly chart with Bars Back settings only load 3 months data. For forex/Futures 5/24 hours instruemts, 3 months data has more bars to display. If I change forex to 8 hours chart, it's the same problem as Stocks 4 hours chart.
The "From date - To date" setting don't have this problem.

arjfca
Posts: 1273
Joined: 23 Nov 2010
Has thanked: 721 times
Been thanked: 222 times

Re: Custom Futures Tick chart causes IB pacing violation

Postby arjfca » 16 Oct 2014

Information from Interactive Brokers regarding Pacing Violation

https://www.interactivebrokers.com/en/s ... ations.htm

wzero
Posts: 142
Joined: 10 Jul 2014
Has thanked: 12 times
Been thanked: 10 times

Re: Custom Futures Tick chart causes IB pacing violation

Postby wzero » 16 Oct 2014

4/8 hours chart problem is nothing related to the pacing violation! No pacing violation pop up during the test. More important, there is no "backfill..." indicate on the chart, meaning fill completed.
These users thanked the author wzero for the post:
Andrew MultiCharts

wzero
Posts: 142
Joined: 10 Jul 2014
Has thanked: 12 times
Been thanked: 10 times

Re: Custom Futures Tick chart causes IB pacing violation

Postby wzero » 05 Nov 2014

It seems the "Bars Back", "Days Back" for custom futures issue still there in Release 2.

User avatar
Andrew MultiCharts
Posts: 1587
Joined: 11 Oct 2011
Has thanked: 931 times
Been thanked: 557 times

Re: Custom Futures Tick chart causes IB pacing violation

Postby Andrew MultiCharts » 06 Nov 2014

wzero wrote:It seems the "Bars Back", "Days Back" for custom futures issue still there in Release 2.

I am sorry for confusion, but it was not supposed to be fixed in MC 9.0 Release 2. It is not yet scheduled for any particular version, please use the "From ... To ...".

Xyzzy
Posts: 156
Joined: 19 Mar 2011
Has thanked: 43 times
Been thanked: 76 times

Re: Custom Futures Tick chart causes IB pacing violation

Postby Xyzzy » 07 Nov 2014

I've had many problems with pacing-violation errors with TWS. For the sake of asking, is there any way -- or could there be any way -- for MultiCharts to throttle the speed of its data requests so that it doesn't trigger the TWS pacing violation?

Currently, my options are:

1. Request data incrementally on a manual basis (e.g., first 200 bars, then 400 bars, then 600 bars, etc.) This is rather tedious.

2. Wait for the "pacing violation" error to pop up and then restart TWS manually. Also tedious.

I'd much rather open a chart and then wait for TWS to fill the chart. If MultiCharts had the ability to throttle the data requests, that might mean that the chart would fill slowly, but without pacing violation errors. I.e., I could order up a chart with 1000 bars of data, get a coffee or lunch, and then come back to a full chart without any errors.

User avatar
Andrew MultiCharts
Posts: 1587
Joined: 11 Oct 2011
Has thanked: 931 times
Been thanked: 557 times

Re: Custom Futures Tick chart causes IB pacing violation

Postby Andrew MultiCharts » 07 Nov 2014

Xyzzy wrote:I've had many problems with pacing-violation errors with TWS. For the sake of asking, is there any way -- or could there be any way -- for MultiCharts to throttle the speed of its data requests so that it doesn't trigger the TWS pacing violation?

We are working on this with IB, but so far there is no ETA when Pacing Violation case should be resolved.
These users thanked the author Andrew MultiCharts for the post (total 2):
wzeroXyzzy

hilbert
Posts: 222
Joined: 17 Aug 2011
Has thanked: 76 times
Been thanked: 64 times

Re: Custom Futures Tick chart causes IB pacing violation

Postby hilbert » 07 Nov 2014

Andrew MultiCharts wrote:
Xyzzy wrote:I've had many problems with pacing-violation errors with TWS. For the sake of asking, is there any way -- or could there be any way -- for MultiCharts to throttle the speed of its data requests so that it doesn't trigger the TWS pacing violation?

We are working on this with IB, but so far there is no ETA when Pacing Violation case should be resolved.

With the special relationship that MC has with IB now (With MC being available inside TWS), I am sure this will get resolved rather sooner.

Thanks for working on this issue.

wzero
Posts: 142
Joined: 10 Jul 2014
Has thanked: 12 times
Been thanked: 10 times

Re: Custom Futures Tick chart causes IB pacing violation

Postby wzero » 13 Oct 2015

Is this problem been fixed in MC 9.1? I downloaded and tested on the Multicharts.Net 9.1 release, the issue is still there. I have waited this problem to be fixed for a year..

User avatar
Henry MultiСharts
Posts: 8503
Joined: 25 Aug 2011
Has thanked: 1213 times
Been thanked: 2731 times

Re: Custom Futures Tick chart causes IB pacing violation

Postby Henry MultiСharts » 19 Oct 2015

Hello wzero,

Pacing Violation is not a MultiCharts issue. This is IB API limitation and that is less likely that it will be removed or softened by IB. That is recommended to use the "From - To" request type to avoid getting the Pacing Violation from IB.

wzero
Posts: 142
Joined: 10 Jul 2014
Has thanked: 12 times
Been thanked: 10 times

Re: Custom Futures Tick chart causes IB pacing violation

Postby wzero » 19 Oct 2015

As the discussion above, the issue is not pacing violation itself, it's about "Bars Back" and "Days Back" for Custom Futures tick charts. Even I set "Bars/Days Back" to a very small value like 1 for Custom Futures tick charts Multicharts will request many many many days back tick data... thus causing pacing violation. The only workaround till now is use "From To" and I have to change it every day.

FutureTrader
Posts: 79
Joined: 28 Mar 2013
Has thanked: 3 times
Been thanked: 4 times
Contact:

Re: Custom Futures Tick chart causes IB pacing violation

Postby FutureTrader » 20 Oct 2015

wzero wrote:As the discussion above, the issue is not pacing violation itself, it's about "Bars Back" and "Days Back" for Custom Futures tick charts. Even I set "Bars/Days Back" to a very small value like 1 for Custom Futures tick charts Multicharts will request many many many days back tick data... thus causing pacing violation. The only workaround till now is use "From To" and I have to change it every day.


Same here, I have custom futures, also I just load 3 days back, it's backfiling years back (as far back as I've added expired futures in QuoteManager ...)
My solution to this is to load your main symbols over the weekend and stick with them as otherwise in live trading you cannot wait for data hours because of pacing violation.


Return to “MultiCharts”