I have recently noticed huge flaws in the IB data. Due to the heavily flawed real-time data collection this morning I loaded up my charts from the previous day and wanted to refresh my charts to get the correct data. I had the chart selected, clicked on the View | Reload | 1 day back menu option. The quote manager showed a data request to the TWS but my chart remained the same still showing incorrect data from the previous days real-time collection. I had to open the quotemanager, select "edit data" to load the last day of minute data and delete it. Only then was my data finally right.
Why didn't the data reload properly when I selected a reload? Is there another way to refresh the last X bars or days that I don't know about?
Interactive Brokers force historical refresh
Re: Interactive Brokers force historical refresh
Hello MidKnight,
I agree with you that IB data feed is not reliable (minute data and daily data have often different closes) and I think MC can enhance the accuracy of their connector about handling real-time data by implementing 5 second bars. But as far as I have seen on other charting softwares, MC (i.e. Quote Manager) is the best at backfilling IB data (may be you have used other charting softwares that I do not know / used). I believe that MC should keep IB as a data feed, even if there are some issues, because other charting vendors are very far from doing as well as MC with TWS (except SC regarding real-time, but not about backfilling). And also, MC supports IB gateway, which can be very convenient sometimes. And having IB as a data feed is also nice to have when one begin trading, because it is free (when we have an account at IB of course).
About the differences about minute data and daily data of IB, I have still not found a definitive solution. But I believe that the ability to build daily bars from minutes data (introduced recently in MC) should be enough for me (i.e. I do not use the daily data of IB). But I understand that it may not be the solution for everyone.
Best regards.
I agree with you that IB data feed is not reliable (minute data and daily data have often different closes) and I think MC can enhance the accuracy of their connector about handling real-time data by implementing 5 second bars. But as far as I have seen on other charting softwares, MC (i.e. Quote Manager) is the best at backfilling IB data (may be you have used other charting softwares that I do not know / used). I believe that MC should keep IB as a data feed, even if there are some issues, because other charting vendors are very far from doing as well as MC with TWS (except SC regarding real-time, but not about backfilling). And also, MC supports IB gateway, which can be very convenient sometimes. And having IB as a data feed is also nice to have when one begin trading, because it is free (when we have an account at IB of course).
About the differences about minute data and daily data of IB, I have still not found a definitive solution. But I believe that the ability to build daily bars from minutes data (introduced recently in MC) should be enough for me (i.e. I do not use the daily data of IB). But I understand that it may not be the solution for everyone.
Best regards.
Last edited by Fabrice on 12 Mar 2014, edited 1 time in total.
Re: Interactive Brokers force historical refresh
Hi Fabrice,
Thanks for the reply. My question is not about getting the daily chart to match the intraday chart. It is strictly about forcing a refresh on the intraday data.
Here is what I have recently noticed.
We collect data for a market in real-time from the IB feed. As we have discussed in other threads, this real-time data is rife with errors. When I do a reload of my intraday chart using the method I describe in the opening post - my chart does not change. It just reloads the same old data with all its errors. To force MC.net to get the "real" data from IB, I needed to manually delete it via the quotemanager and then load the chart.
Thanks for the reply. My question is not about getting the daily chart to match the intraday chart. It is strictly about forcing a refresh on the intraday data.
Here is what I have recently noticed.
We collect data for a market in real-time from the IB feed. As we have discussed in other threads, this real-time data is rife with errors. When I do a reload of my intraday chart using the method I describe in the opening post - my chart does not change. It just reloads the same old data with all its errors. To force MC.net to get the "real" data from IB, I needed to manually delete it via the quotemanager and then load the chart.
- Henry MultiСharts
- Posts: 9165
- Joined: 25 Aug 2011
- Has thanked: 1264 times
- Been thanked: 2958 times
Re: Interactive Brokers force historical refresh
Hello MidKnight,
Due to IB Pacing Violation limitation MultiCharts stores realtime IB quotes in a cache until the restart of the platform. When you reload a chart in order to avoid pacing violation the data is taken from this cache.
If you want the data to be directly loaded from IB servers, please disable the option “Utilization of cache for real-time price data” in Quote Manager->Tools->Data sources->IB-> Settings. To make this setting come into effect you need to restart MultiCharts and all of its applications.
Due to IB Pacing Violation limitation MultiCharts stores realtime IB quotes in a cache until the restart of the platform. When you reload a chart in order to avoid pacing violation the data is taken from this cache.
If you want the data to be directly loaded from IB servers, please disable the option “Utilization of cache for real-time price data” in Quote Manager->Tools->Data sources->IB-> Settings. To make this setting come into effect you need to restart MultiCharts and all of its applications.
Re: Interactive Brokers force historical refresh
Thanks for the reply Henry. This sparks some further questions.
1) If you store it in a cache and the user chooses to reload - the chart is just loading the same already defective data - so whats the point of the reload feature?
2) I have experienced collecting data in real-time. Closing MC.net for the day. Opening MC.net the next day before the market has opened. Reloading the chart 1 day back -- and still the chart doesn't refresh from IB servers. You say the real-time data is stored in a cache. This example has not collected any real-time data yet. So even though I hit "reload", it is only reloading my data from the local database - again, whats the point of the feature reload then?
I could see the loaded chart price related data being a great idea to keep in the cache so subsequent studies applied only refer to the cache. But, in my opinion, from a users point of view. The reload feature (or something similar other than the checkbox you describe) should be available to force a reload from the data server.
1) If you store it in a cache and the user chooses to reload - the chart is just loading the same already defective data - so whats the point of the reload feature?
2) I have experienced collecting data in real-time. Closing MC.net for the day. Opening MC.net the next day before the market has opened. Reloading the chart 1 day back -- and still the chart doesn't refresh from IB servers. You say the real-time data is stored in a cache. This example has not collected any real-time data yet. So even though I hit "reload", it is only reloading my data from the local database - again, whats the point of the feature reload then?
I could see the loaded chart price related data being a great idea to keep in the cache so subsequent studies applied only refer to the cache. But, in my opinion, from a users point of view. The reload feature (or something similar other than the checkbox you describe) should be available to force a reload from the data server.
- JoshM
- Posts: 2195
- Joined: 20 May 2011
- Location: The Netherlands
- Has thanked: 1544 times
- Been thanked: 1565 times
- Contact:
Re: Interactive Brokers force historical refresh
The cache is persistent over multiple MC sessions, and even remains when upgrading MultiCharts. See the `TSCACHE.GDB` file in `C:\ProgramData\TS Support\MultiCharts .NET64\Databases`.2) I have experienced collecting data in real-time. Closing MC.net for the day. Opening MC.net the next day before the market has opened. Reloading the chart 1 day back -- and still the chart doesn't refresh from IB servers. You say the real-time data is stored in a cache. This example has not collected any real-time data yet. So even though I hit "reload", it is only reloading my data from the local database - again, whats the point of the feature reload then?
I agree. There's little sense in reloading from cache, especially since the whole point of the reload function is to get fresh data.I could see the loaded chart price related data being a great idea to keep in the cache so subsequent studies applied only refer to the cache. But, in my opinion, from a users point of view. The reload feature (or something similar other than the checkbox you describe) should be available to force a reload from the data server.
IB users probably know, or otherwise quickly learn, that historical data requests can lead to pacing violation and are thus better advised not to reload dozens of days at a time. I don't see how reloading from cache is a helpful option for MC users, since they are probably already aware of the limitations and so can adjust their reload request accordingly.
Two questions if I may:
* Does the reload function also reload from cache for other data feeds?
* Can the Wiki page for reloading charts be updated with this information? Incomplete wiki pages are sometimes worse than no wiki page, especially when using functions programmatically.
Re: Interactive Brokers force historical refresh
I really agree with JoshM about the need to better document how to configure the IB connector.
I have been using MC and IB for several years now, and I must say that I am still not sure if the tick chart of yesterday is the real-time or a fresh reload of the morning - and I have even the feeling that this behavior has changed from release to release, but it is just a feeling. This does not prevent me from trading, so I don't mind too much, but it would be nice to fully understand when the IB connector automatically reloads a tick chart, and why.
Regards.
I have been using MC and IB for several years now, and I must say that I am still not sure if the tick chart of yesterday is the real-time or a fresh reload of the morning - and I have even the feeling that this behavior has changed from release to release, but it is just a feeling. This does not prevent me from trading, so I don't mind too much, but it would be nice to fully understand when the IB connector automatically reloads a tick chart, and why.
Regards.
- Henry MultiСharts
- Posts: 9165
- Joined: 25 Aug 2011
- Has thanked: 1264 times
- Been thanked: 2958 times
Re: Interactive Brokers force historical refresh
That is a different cache. TSCACHE.GDB is a file that stores the data unpacked from the database and it is not wiped upon restart.The cache is persistent over multiple MC sessions, and even remains when upgrading MultiCharts. See the `TSCACHE.GDB` file in `C:\ProgramData\TS Support\MultiCharts .NET64\Databases`.
The cached IB realtime data is storred in memory cache and is cleared upon all processes restart.
I have already described how to disable the realtime cache for IB in the post #4 above.I agree. There's little sense in reloading from cache, especially since the whole point of the reload function is to get fresh data. IB users probably know, or otherwise quickly learn, that historical data requests can lead to pacing violation and are thus better advised not to reload dozens of days at a time. I don't see how reloading from cache is a helpful option for MC users, since they are probably already aware of the limitations and so can adjust their reload request accordingly.I could see the loaded chart price related data being a great idea to keep in the cache so subsequent studies applied only refer to the cache. But, in my opinion, from a users point of view. The reload feature (or something similar other than the checkbox you describe) should be available to force a reload from the data server.
No.Two questions if I may:
* Does the reload function also reload from cache for other data feeds?
The page for IB data provider has been udpated with this info.* Can the Wiki page for reloading charts be updated with this information? Incomplete wiki pages are sometimes worse than no wiki page, especially when using functions programmatically.