ib data fill

Questions about MultiCharts and user contributed studies.
O66
Posts: 143
Joined: 28 Nov 2006
Location: Netherlands
Has thanked: 2 times
Been thanked: 3 times

ib data fill

Postby O66 » 10 Sep 2007

i did upgrade to latest beta 2.1.976.1109
after waiting more then a hour to get some charts loaded i started quote manager. the realtime status for symbol was Offline. (imo it was that all the time but i still had charts in previous versions, never understood what realtime status was) i did right click and choose connect, immediately data was comming in. I closed all my workspaces and started a new 2 min er2 chart with data range set to 60 bars back.
in qm i see now 2871 received but i still dont have a chart.
the chart message is: establishing connection .....


what is going on? What is it what im doing wrong?
what is realtime status and where is it used for?
how can i get charts ? ( i even dit set datarange to 1 bar)

thanks

User avatar
Marina Pashkova
Posts: 2758
Joined: 27 Jul 2007

Re: ib data fill

Postby Marina Pashkova » 10 Sep 2007

i did upgrade to latest beta 2.1.976.1109
after waiting more then a hour to get some charts loaded i started quote manager. the realtime status for symbol was Offline. (imo it was that all the time but i still had charts in previous versions, never understood what realtime status was) i did right click and choose connect, immediately data was comming in. I closed all my workspaces and started a new 2 min er2 chart with data range set to 60 bars back.
in qm i see now 2871 received but i still dont have a chart.
the chart message is: establishing connection .....


what is going on? What is it what im doing wrong?
what is realtime status and where is it used for?
how can i get charts ? ( i even dit set datarange to 1 bar)


Online symbol status in QuoteManager is needed if you want to collect data for a symbol without plotting the chart. The data will be going directly into the storage without you having to plot a chart.

If you see a number of quotes received but no chart, it means that not all the data you have requested have been collected yet. A chart is only plotted when all the data you requested have been received.

O66
Posts: 143
Joined: 28 Nov 2006
Location: Netherlands
Has thanked: 2 times
Been thanked: 3 times

Postby O66 » 10 Sep 2007

datarange is set to 1 bar!
new data is comming in
maybe an ib backfill problem ?

when is mc changed so it starts showing charts without loading all the data first ?

User avatar
Marina Pashkova
Posts: 2758
Joined: 27 Jul 2007

Postby Marina Pashkova » 10 Sep 2007

There have been problems at ib. Their datafeed is now working very slowly.

O66
Posts: 143
Joined: 28 Nov 2006
Location: Netherlands
Has thanked: 2 times
Been thanked: 3 times

Postby O66 » 10 Sep 2007

finally have charts
" unchecked i want to download missing data" , MC was waiting for IB
when will this be changed in MC?

Clifford Farrar
Posts: 1
Joined: 25 Aug 2007

Postby Clifford Farrar » 10 Sep 2007

Hello,

I see the same problem when I installed the latest version. My test chart (NQU7) never came up. I closed and reopened but as 066 mentioned, unchecked the missing data checkbox and the chart came up.

User avatar
Marina Pashkova
Posts: 2758
Joined: 27 Jul 2007

Postby Marina Pashkova » 11 Sep 2007

finally have charts
" unchecked i want to download missing data" , MC was waiting for IB
when will this be changed in MC?


What would you like to have changed?

"I want to download missing data" is used if you have periods for which there was no data and you want to fill them. Of course, if the data feed has problems with sending quotes it might take a long time. This is not the MC problem.

If you are referring to the MC way of plotting charts, when a chart is only plotted once all the requested data have been received - this will be changed in future MC versions.

traderstuff
Posts: 68
Joined: 24 Jul 2005

Postby traderstuff » 14 Sep 2007

I am not sure if this should be posted here or with a new title -"BUG" .

Am positive something similar has been posted before concerning an older version of MC.

Using latest MC ver#1109 under Win XP Pro,I did not change the IB TWS (ver874.4 (July-2007) to exit in the a.m.,so last night it shut down at 11:59 p.m.

At about 4 a.m. this morning,I reset IB tws,and MC connected Realtime to the 20 current futures contracts that are listed in QM.

The problems are that QM or whatever part of MC 'should have' requested historical data fill in 'did Not do so.

QM Did show Realtime data was being received,and the 'new data was being charted,-Without the missing 4 hours plus of historical data

I had to close QM and MC,and then restart them to start the historical data collection with IB.
And of course the infamous IB data curse of Numerous ''pacing violation" kicked in.. [wasn't that suppose to have been fixed in this release of QM?]

I do have the request to fill in missing data checked,and it was checked when the above happened.

It is now one hour and forty six minutes since I restarted MC and QM and the data is still being summoned from IB. Also have a cable internet connection,so 'speed' on this end should not be a problem.

Data being used is mostly volume charts,with a couple of tick
charts.
Length varies from the least of 3 days and greatest of 8 days.

However that should not be relevent as the data was supposedly current up to 11:59 p.m., about 4 hours missing when initial fill in data requested.

User avatar
Marina Pashkova
Posts: 2758
Joined: 27 Jul 2007

Postby Marina Pashkova » 14 Sep 2007

Hello,

Weren't gaps filling for all the resolutions? Usually, if you have minute resolutions the gaps won't start filling automatically: you need to close and open the workspace or the chart again. For tick resolutions gaps should be filled automatically.

As for the pacing violation, it is not the issue of speed on your end nor it is the issue of MultiCharts: it is related to the IB's processing data requests. MultiCharts only shows the 'pacing violation' message to let the users know that IB cannot speedily process their requests. In future, we will give the users an option to reduce the number of requests by changing resolution (for example, from 1 second to 5 seconds)

traderstuff
Posts: 68
Joined: 24 Jul 2005

Postby traderstuff » 18 Sep 2007

Hello,

Weren't gaps filling for all the resolutions? Usually, if you have minute resolutions the gaps won't start filling automatically: you need to close and open the workspace or the chart again. For tick resolutions gaps should be filled automatically.

As for the pacing violation, it is not the issue of speed on your end nor it is the issue of MultiCharts: it is related to the IB's processing data requests. MultiCharts only shows the 'pacing violation' message to let the users know that IB cannot speedily process their requests. In future, we will give the users an option to reduce the number of requests by changing resolution (for example, from 1 second to 5 seconds)

********************

No,gaps were not filling,but current prices were.

Since my original post, I had a similar occurence but it was because I did not change the IB tws to change at 11:59 PM instead of 11:59 AM.
This time MC DID fill in the missing data!

Perhaps the change you mention will allow users to avoid the 'PACING' violation - but have tests been run by support to find the max fast download from IB historical data,say,for instance,the ES contract,regardless of pacing issues or not?

Nick
Posts: 490
Joined: 04 Aug 2006
Has thanked: 4 times
Been thanked: 24 times

Postby Nick » 19 Sep 2007

Hello,

Weren't gaps filling for all the resolutions? Usually, if you have minute resolutions the gaps won't start filling automatically: you need to close and open the workspace or the chart again. For tick resolutions gaps should be filled automatically.

As for the pacing violation, it is not the issue of speed on your end nor it is the issue of MultiCharts: it is related to the IB's processing data requests. MultiCharts only shows the 'pacing violation' message to let the users know that IB cannot speedily process their requests. In future, we will give the users an option to reduce the number of requests by changing resolution (for example, from 1 second to 5 seconds)


I know IB were absolutely &^%$ at publishing the protocol for HDMS server requests. However if you pin them down and follow thier protocol it is possible to avoid pacing violations all together. Its simply a question of only makeing X requests every Y minutes with at least Z seconds between requests. If you get the correct value for XY&Z you get maximum data throughput with not a single pacing violation.

I posted the values and details of the protocol here when they first introduced client side throttling. A colleague of mine had managed to get them out of IB after a brief dialogue (they gave out wrong values initially). Maybe they have changed the protocol again but that seems unlikely.

Giving the customer control over Z is really not the way to go and dosent help as there are two separate limits. Other applications manage this with maximum throughput and no pacing violations so I am sure its doable! Of course getting info out of IB is like getting blood out of a stone!!

Cheers.
Nick.

User avatar
Marina Pashkova
Posts: 2758
Joined: 27 Jul 2007

Postby Marina Pashkova » 19 Sep 2007

Hi again Nick,

We have explored the issue on many occasions and came to the conclusion that the current implementation of data requesting is optimal. We have examined the work of other platforms and with them the issue is simply that the data is received slowly and you never know that there is a lag.

The suggestion above would not change the overall time of getting the requested data. The time will always be the same whether a lot of data is requested and then you have to wait for this big chunk to arrive or when small sections are plotted. In either case the resulting time would be the same.

You could also check with the IB website and read how their datafeed functions.

User avatar
Marina Pashkova
Posts: 2758
Joined: 27 Jul 2007

Postby Marina Pashkova » 19 Sep 2007

Hi again Nick,

We have explored the issue on many occasions and came to the conclusion that the current implementation of data requesting is optimal.

The suggestion above would not change the overall time of getting the requested data. The time will always be the same whether a lot of data is requested and then you have to wait for this big chunk to arrive or when small sections are plotted. In either case the resulting time would be the same.

You could also check with the IB website and read how their datafeed functions.


Return to “MultiCharts”