Interactive Brokers force historical refresh

Questions about MultiCharts .NET and user contributed studies.
MidKnight
Posts: 343
Joined: 12 Aug 2012
Has thanked: 123 times
Been thanked: 56 times

Interactive Brokers force historical refresh

Postby MidKnight » 10 Mar 2014

I have recently noticed huge flaws in the IB data. Due to the heavily flawed real-time data collection this morning I loaded up my charts from the previous day and wanted to refresh my charts to get the correct data. I had the chart selected, clicked on the View | Reload | 1 day back menu option. The quote manager showed a data request to the TWS but my chart remained the same still showing incorrect data from the previous days real-time collection. I had to open the quotemanager, select "edit data" to load the last day of minute data and delete it. Only then was my data finally right.

Why didn't the data reload properly when I selected a reload? Is there another way to refresh the last X bars or days that I don't know about?

Fabrice
Posts: 182
Joined: 14 Jun 2011
Has thanked: 44 times
Been thanked: 30 times

Re: Interactive Brokers force historical refresh

Postby Fabrice » 11 Mar 2014

Hello MidKnight,

I agree with you that IB data feed is not reliable (minute data and daily data have often different closes) and I think MC can enhance the accuracy of their connector about handling real-time data by implementing 5 second bars. But as far as I have seen on other charting softwares, MC (i.e. Quote Manager) is the best at backfilling IB data (may be you have used other charting softwares that I do not know / used). I believe that MC should keep IB as a data feed, even if there are some issues, because other charting vendors are very far from doing as well as MC with TWS (except SC regarding real-time, but not about backfilling). And also, MC supports IB gateway, which can be very convenient sometimes. And having IB as a data feed is also nice to have when one begin trading, because it is free (when we have an account at IB of course).

About the differences about minute data and daily data of IB, I have still not found a definitive solution. But I believe that the ability to build daily bars from minutes data (introduced recently in MC) should be enough for me (i.e. I do not use the daily data of IB). But I understand that it may not be the solution for everyone.

Best regards.
Last edited by Fabrice on 12 Mar 2014, edited 1 time in total.

MidKnight
Posts: 343
Joined: 12 Aug 2012
Has thanked: 123 times
Been thanked: 56 times

Re: Interactive Brokers force historical refresh

Postby MidKnight » 11 Mar 2014

Hi Fabrice,

Thanks for the reply. My question is not about getting the daily chart to match the intraday chart. It is strictly about forcing a refresh on the intraday data.

Here is what I have recently noticed.

We collect data for a market in real-time from the IB feed. As we have discussed in other threads, this real-time data is rife with errors. When I do a reload of my intraday chart using the method I describe in the opening post - my chart does not change. It just reloads the same old data with all its errors. To force MC.net to get the "real" data from IB, I needed to manually delete it via the quotemanager and then load the chart.

User avatar
Henry MultiСharts
Posts: 9165
Joined: 25 Aug 2011
Has thanked: 1264 times
Been thanked: 2957 times

Re: Interactive Brokers force historical refresh

Postby Henry MultiСharts » 13 Mar 2014

Hello MidKnight,

Due to IB Pacing Violation limitation MultiCharts stores realtime IB quotes in a cache until the restart of the platform. When you reload a chart in order to avoid pacing violation the data is taken from this cache.

If you want the data to be directly loaded from IB servers, please disable the option “Utilization of cache for real-time price data” in Quote Manager->Tools->Data sources->IB-> Settings. To make this setting come into effect you need to restart MultiCharts and all of its applications.

MidKnight
Posts: 343
Joined: 12 Aug 2012
Has thanked: 123 times
Been thanked: 56 times

Re: Interactive Brokers force historical refresh

Postby MidKnight » 13 Mar 2014

Thanks for the reply Henry. This sparks some further questions.

1) If you store it in a cache and the user chooses to reload - the chart is just loading the same already defective data - so whats the point of the reload feature?

2) I have experienced collecting data in real-time. Closing MC.net for the day. Opening MC.net the next day before the market has opened. Reloading the chart 1 day back -- and still the chart doesn't refresh from IB servers. You say the real-time data is stored in a cache. This example has not collected any real-time data yet. So even though I hit "reload", it is only reloading my data from the local database - again, whats the point of the feature reload then?

I could see the loaded chart price related data being a great idea to keep in the cache so subsequent studies applied only refer to the cache. But, in my opinion, from a users point of view. The reload feature (or something similar other than the checkbox you describe) should be available to force a reload from the data server.

User avatar
JoshM
Posts: 2195
Joined: 20 May 2011
Location: The Netherlands
Has thanked: 1544 times
Been thanked: 1565 times
Contact:

Re: Interactive Brokers force historical refresh

Postby JoshM » 13 Mar 2014

2) I have experienced collecting data in real-time. Closing MC.net for the day. Opening MC.net the next day before the market has opened. Reloading the chart 1 day back -- and still the chart doesn't refresh from IB servers. You say the real-time data is stored in a cache. This example has not collected any real-time data yet. So even though I hit "reload", it is only reloading my data from the local database - again, whats the point of the feature reload then?
The cache is persistent over multiple MC sessions, and even remains when upgrading MultiCharts. See the `TSCACHE.GDB` file in `C:\ProgramData\TS Support\MultiCharts .NET64\Databases`.
I could see the loaded chart price related data being a great idea to keep in the cache so subsequent studies applied only refer to the cache. But, in my opinion, from a users point of view. The reload feature (or something similar other than the checkbox you describe) should be available to force a reload from the data server.
I agree. There's little sense in reloading from cache, especially since the whole point of the reload function is to get fresh data.

IB users probably know, or otherwise quickly learn, that historical data requests can lead to pacing violation and are thus better advised not to reload dozens of days at a time. I don't see how reloading from cache is a helpful option for MC users, since they are probably already aware of the limitations and so can adjust their reload request accordingly.

Two questions if I may:
* Does the reload function also reload from cache for other data feeds?
* Can the Wiki page for reloading charts be updated with this information? Incomplete wiki pages are sometimes worse than no wiki page, especially when using functions programmatically.

Fabrice
Posts: 182
Joined: 14 Jun 2011
Has thanked: 44 times
Been thanked: 30 times

Re: Interactive Brokers force historical refresh

Postby Fabrice » 14 Mar 2014

I really agree with JoshM about the need to better document how to configure the IB connector.

I have been using MC and IB for several years now, and I must say that I am still not sure if the tick chart of yesterday is the real-time or a fresh reload of the morning - and I have even the feeling that this behavior has changed from release to release, but it is just a feeling. This does not prevent me from trading, so I don't mind too much, but it would be nice to fully understand when the IB connector automatically reloads a tick chart, and why.

Regards.

User avatar
Henry MultiСharts
Posts: 9165
Joined: 25 Aug 2011
Has thanked: 1264 times
Been thanked: 2957 times

Re: Interactive Brokers force historical refresh

Postby Henry MultiСharts » 24 Mar 2014

The cache is persistent over multiple MC sessions, and even remains when upgrading MultiCharts. See the `TSCACHE.GDB` file in `C:\ProgramData\TS Support\MultiCharts .NET64\Databases`.
That is a different cache. TSCACHE.GDB is a file that stores the data unpacked from the database and it is not wiped upon restart.

The cached IB realtime data is storred in memory cache and is cleared upon all processes restart.
I could see the loaded chart price related data being a great idea to keep in the cache so subsequent studies applied only refer to the cache. But, in my opinion, from a users point of view. The reload feature (or something similar other than the checkbox you describe) should be available to force a reload from the data server.
I agree. There's little sense in reloading from cache, especially since the whole point of the reload function is to get fresh data. IB users probably know, or otherwise quickly learn, that historical data requests can lead to pacing violation and are thus better advised not to reload dozens of days at a time. I don't see how reloading from cache is a helpful option for MC users, since they are probably already aware of the limitations and so can adjust their reload request accordingly.
I have already described how to disable the realtime cache for IB in the post #4 above.
Two questions if I may:
* Does the reload function also reload from cache for other data feeds?
No.
* Can the Wiki page for reloading charts be updated with this information? Incomplete wiki pages are sometimes worse than no wiki page, especially when using functions programmatically.
The page for IB data provider has been udpated with this info.


Return to “MultiCharts .NET”