Quotemanager and IB . . . . .

1.)i understand that IB as a datafeed is not really competitive
however i really dont understand how quotemanager is caching the
collected data - when backfilling i often if not even always recognize that
its backfilling days (sometimes many of them !..) that already are or should be cached in QT.

2.)that means when opening a workspace with just 4 symbols its like watching paint dry until you see any chart plotted ...that again means that before starting the day it should be considered to start MC 2 hours earlier so that it can collect and backfill until its happy ....

3.)also not very efficiently is that QT is not able to use its cache to plot a chart immediatly where the data is already collected .what i mean by this is the following:
imagine you have opened for example eurusd forex charts - lets say 5min and 15 and 60 min intervals in other workspaces that already are running .
to make it easier i call them here "A" and "B" workspaces.

then there is a workspace where QT still has to collect the data for some charts that have to be plotted .i call it "C" workspace

now you open another workspace where one chart contains eurusd in 5 min interval and charts from other datasource like IQFEED - i call it "D" workspace

ALTHOUGH its definatly the fact that eurusd is up and running in other workspaces - ("A" and "B") it wont happen that the [5min eurusd]chart in the new opened workspace ("D") will plot until QT managed to completely collect the data for the workspace of "C" which like i stated on top can take ages ....

4.)the other thing is that maybe it would be faster to backfill if QT wouldnt always want to collect minute data - sometimes it would be OK just to have the interval collected that you want to plot -for example 15 minute interval.
(in terms of backtesting capability in later builds of MC i understand that this is not really good but then again backtesting with IB data is a thing for itself..)

5.)when using IQFEED with IB together sometimes opening up too many workspaces causes my PC to freeze up

6.)last but not least - MC has great potential thanks for developing it and keep up the great work !

regards
NEUTRINO

The problem with IB is that QuoteManager tries ro re-request data from the server even if there is no data - like, in case there is a difference between Quote Manager's session settings and the server.

For instance, if the symbol is traded 8 hours per day and QuoteManager has 24 hour sessions set, it will try re-requesting 16 hour's wort of data ever time - of you watch the log closely you'll see that it tries to redownload the same data.

Were going to add an option for diabling this; presently what you can try is use
Online data server mode (File-Preferences-Data Server).

I think the better solution would be to allow quotemanager to schedule downloads of data for each symbol.

So we can backfill say the previous day's data when market close.
That way, users can leave their machine open and it will automatically refresh from source.

The next time when users open up MC, it opens immediately without having to backfill again.

The problem with IB is that QuoteManager tries ro re-request data from the server even if there is no data - like, in case there is a difference between Quote Manager's session settings and the server.

For instance, if the symbol is traded 8 hours per day and QuoteManager has 24 hour sessions set, it will try re-requesting 16 hour's wort of data ever time - of you watch the log closely you'll see that it tries to redownload the same data.

Were going to add an option for diabling this; presently what you can try is use
Online data server mode (File-Preferences-Data Server).

Thanks for the feedback, but this is not the soluton we're pursuing, our goal is to get the IB data feed to function properly as other datafeeds do.

I for one, think scheduling downloads is a great idea.

I think the feature about scheduling downloads is not specific to IB or any other data feed. It has nothing to do with MC's need to get its software to work with IB feed properly. Even though it can solve some of the problems.

Scheduling downloads will allow users to
1. Get their data updated even though they did not trade for a couple of days.
As long as they keep their pc running, it will download/refresh data on their own and the next time they want to trade, the data is up to date.

2. It will also allow data to be updated when we do not have the chart open. Currently if we want to monitor 50 symbols and we want the data updated and refresh from backfill constantly, all 50 symbols must be open up in some charts.
This is silly because we may only actively monitor 5.

I am sure there are many other advantages that other users can think of.

Thanks for the feedback, but this is not the soluton we're pursuing, our goal is to get the IB data feed to function properly as other datafeeds do.

This thread looks like my problem here Nov 08, 2006:
Post subject: Intermittent IB historical tick
http://forum.tssupport.com/viewtopic.php?t=2539

But what happend now, i am not clear about the handling with IB-feed and backfill.

How to handle realtime and historical data in MultiCharts?

It is not clear for me, why historical data can not be handled in QuoteManager.
The concept of MultiCharts is that charts should handle historical data, but this is not possible because:

1. it take hours to download historical data
2. the data get lost if changing the timframe in chart

So, MultiCharts can not be used in this status with data from IB. This is my poor result. Very sadly because it was my big hope that MultiCharts ist the software which does all the things I want to do...

Why not handle all the data in QuoteManager?

Its the problem of merging the smallest timframe of 1 second from historical data with the snapshot-ticks of realtime data?

What is the plan of tssupport.com to handle this problem? Or is there a possibility now?

All the data is managed in QuoteManager - I really did not understand the point of question.

Downloading may be taking unacceptably long if the DataServer is set to "I want to download missing data" - in this case it will always try to fill every gap, and if data to fill it is not available, the chart will never show up. If you look up the QuoteManager log you'll see endless requests for data to cover that period of time.

If, when changing the timeframe you include a gap that cannot be filled, the same will happen - remove "I want to download missing data" to receive all the data you can get from the source and use it.

All the data is managed in QuoteManager - I really did not understand the point of question.

Downloading may be taking unacceptably long if the DataServer is set to "I want to download missing data" - in this case it will always try to fill every gap, and if data to fill it is not available, the chart will never show up. If you look up the QuoteManager log you'll see endless requests for data to cover that period of time.

If, when changing the timeframe you include a gap that cannot be filled, the same will happen - remove "I want to download missing data" to receive all the data you can get from the source and use it.

if data to fill it is not available, the chart will never show up.

won't you consider this a deficiency in MC ?
a little routine can easily circumvent this. Say, post a flag on the chart to show that there is a data gap there?

I have been using 10 wsp for a month, and everything worked fine.
This morning after I rebooted the computer, not a chart is drawing.
This is very frustrating. I was waiting for the chart to show up, not knowing what went wrong, I rebooted the computer again, and more waiting, while the market was moving away.

All the data is managed in QuoteManager - I really did not understand the point of question.

Maybe this is an important point to prefent you from more annoying questions to this theme.

I understand now the problem is, that in chart is a data range setting (which is available from IB), but WITH a compression, that is not available from IB for this data range!

For example, a data range of 1 year is available WITH 1 day compression, but not with 1 min, or 1 hour compression.
I did not know this. I had this setting with the euribor on liffe and was wondering why the chart was loading all the time I changed the timframe in the chart. And deduced that my 1 minute data that i loaded before, was not stored in MC. Because MC made the impression for me that all data of the range are loaded in 1 minute compression. I did not realised, that a few months in 1 minute data compression was not loaded.

Do you understand my point now? People with datafeed from IB must know this, else they think MC is queer or something like this...

Generally speaking, if there's something amiss with IB data, it makes sense to look up the TWS website or contact their support service.

Things with data limits etc. are liable to change whenever the data provider thinks fit, it is really not our duty to tell things about datafeeds things that may become outdated.

Generally speaking, if there's something amiss with IB data, it makes sense to look up the TWS website or contact their support service.

Things with data limits etc. are liable to change whenever the data provider thinks fit, it is really not our duty to tell things about datafeeds things that may become outdated.

Not amiss with IB-Data, amiss with MultiCharts handling Ib-Data. And you will have a lot of amiss with unsatisfied customers from IB, if you dont want to understand.
It would be better to cancel the IB-Datafeed feature. There is no advantage to customers, to have a nice software architecture (and MultiCharts is a realy nice architecture) if it is not usable with the brokerfeed of IB.
My wish is:
1. give this hints of a few sentences to the customers of IB-Datafeed, you dont need to do IB-customer support, but what use do you have if the people think the problem are coming from MultiCharts?

You should a little bit grateful, that i did this ravel of experiences of data range and time compression with IB data...
I did it because i would like to use MultiCharts....

Dear templ -

We're sure that our handling of IB data is the best available anywhere outside IB itself.
There is no fault of ours if IB has issues in its data stream - if this happens, do not use the "I want to download missing data" in the Preferences->Data server mode, and the chart will be plotted from cache quickly, skipping the problems.

If there was a way to retrieve data that is broken or does not exist, we would have implemented it long ago.

Dear templ -

We're sure that our handling of IB data is the best available anywhere outside IB itself.
There is no fault of ours if IB has issues in its data stream - if this happens, do not use the "I want to download missing data" in the Preferences->Data server mode, and the chart will be plotted from cache quickly, skipping the problems.

If there was a way to retrieve data that is broken or does not exist, we would have implemented it long ago.

Alex, I would like to add something to this discussion about how to handle missing IB data. I think that templ has tried to indicate that from the user's point of view that it makes more sense to display what data is available, rather than just having to choose between (a) endlessly 'waiting' for (any, unknown) 'missing' data, or just (b) NEVER even ask for data that may be now available, but once was missing. The principle of providing the user with better 'status' of missing/not-missing data by displaying what IS available (and then FLAGGING the 'holes' in the missing bars) is a much more effective way to handle the IB datafeed issue. Why? Because it gives the user information that can be adaptively handled in a gracefull and time-efficient manner.

I.E., Not knowing for *certain* whether *this is the day* that IB's datafeed is a problem, AND therefore being 'forced' to discount the possibility that *maybe* the datafeed is ok today (by doing what you suggeted),...... mmmmm, does this sound like a 'solution'? Mr templ had a good idea because it addresses the real issue of how important it is to the user to have 'something' useful, rather than 'nothing', in the way of status information for the IB (missing?) data.

It shouldn't take much more effort to implement the idea of templ. It is already being logged whether there is 'missing' data, and so rather than the QuoteManager program responding to this status the way it does now, it could display what ever data is available, and adding some kind of visual place holder for the missing data. Then the user gets something he can deal with in a graceful manner, without so much frustration .

Dear denizen2 -

Things that seem convenient turn can out extremely difficult to implement.

We tried once, and analyzing empty bars turned out to be a problem that would take too much effort to solve. Really, if you're using the computer for charting and do not shut it down during trading hours, holes should be rare, so you could find them by looking at a chart. Could you please name an example of such working gap flagging?

Dear denizen2 -

Things that seem convenient turn can out extremely difficult to implement.

We tried once, and analyzing empty bars turned out to be a problem that would take too much effort to solve. Really, if you're using the computer for charting and do not shut it down during trading hours, holes should be rare, so you could find them by looking at a chart. Could you please name an example of such working gap flagging?

I admit that implementation of software is almost *never* a trivial exercise But, isn't there some simple way to just display whatever data IS available, at any given time, rather than waiting for *every* requested data bar to be delivered, .... for *every* symbol on a given plot?

I.e., The 'flagging' of the 'missing' data is just one part of the proposed approach. More fundamental aspect of the 'idea' is that the displayed data is 'asyncronously' delivered, i.e., without waiting for more than a certain amount of time, or a certain minimum amount of data (say, 100-300 bars).

The same principle applies to any data-vendor (not just IB). I use eSignal, and some times I have to wait a very long time for my plot to be displayed. I thinkit is because I have 10min, 1HR, and 4Hr bars of the same symbol all being requested at the same time (for maybe a year's worth of data). I would much prefer to see the first 3-months of data displayed quickly, and then allow the rest of the later data be backfilled, as it is made available. At least I can see the progress being made by it being displayed, rather than waiting for 10minutes or longer for everything to be received.

So, would an asyncronous display-data-delivery approach be more easily implemented without any requirement for 'flaggin' missing data? Maybe it would be easy enough to just include a status 'light' on the bottom of the screen that says 'data not completed (yet)', but not worry about a space-filler type flag on the missing bars?

Thanks, we're considering this - and maybe even making asynchronous, "chart-as-loaded" plotting an option.

Thanks, we're considering this - and maybe even making asynchronous, "chart-as-loaded" plotting an option.

Thanks, Alex, that sounds great

I just had another 'thought' about something else that will come up when cnsidering design and implementation of 'chart-as-loaded' feature-option: What should happen to those indicators that are based on the the data being loaded?

Should the indicators NOT be calculated and displayed until ALL of the data requested has been loaded? I believe that this is the current scheme used, but what if there is a long-wait for, say, the last half of, the data?

Do you have to wait until everything is loaded before the indicators are calculated, even though you have 'sufficient' data for a screen full of calculated indicators on the screen?

And then there is the issue also of how often to 'update' the calculated indicators during the chart-as-loaded scheme? It would not be practical to re-calculate the indicator(s) every time some small number of data points are loaded, right? So maybe the most reasonable approach to that issue would be to determine how many data bars are represented on the screen at the current screen-settings, and do the same for each symbol on the same plot, and then use that number(s) as the 'buffer-block-size' for each symbol. Then it could be assumed that the screeen and calculated indicators will be updated each time a 'new' buffer-block' of data has been delivered .

Futhermore, I would suggest that the status line at the top of the plot be used to show if the data is all received, or some is still pending. That could be done by maybe blinking the status line, and also adding a special progress indicator field that says what percentage has been loaded. Also, the same status line could be used to indicate if anything longer than a typical session-closed period of time is 'missing', but just ignore any missing bars of smaller length?

So thanks again for considering these ideas for future improvements of a great charting software.

An indicator could be applied to an incomplete chart, but with one condition - as new data arrives it will be recalculated so when the symbol finally loads, the indicator can end up looking quite different from when it started.

As to the progress indicator in the status line - we have no power over the data source. Such an indicator, showing percents or a color bar, could work if data downloading was always a stable process, at least as predictable as say downloading a file over a HTTP connection.

In reality, if data is delayed because a gap/bad tick occurred (the most common reason for delays/repeated queries) this will not work, same if the data source stopped delivering data for any reason.

As to the progress indicator in the status line - we have no power over the data source. Such an indicator, showing percents or a color bar, could work if data downloading was always a stable process, at least as predictable as say downloading a file over a HTTP connection.

In reality, if data is delayed because a gap/bad tick occurred (the most common reason for delays/repeated queries) this will not work, same if the data source stopped delivering data for any reason.

Alex... what am I missing here? Your statment is not making any sense to me If you know how much data is requested, N-REQ, and you can always count how many bars have been received at any current time in the process, N-Recvd, then the "progress" status is simply 100* (N-REQ minus N-RECVD)/ N-REQ when expressed as a percentage, right? It doesn't matter WHY the data is not yet delivered, if all that is beng expressed in the status is HOW MUCH data has been received. All of the issues about bad-ticks and repeated request or delays..... well that pertains to why the data is not yet be fully delivered, but has nothing to do with the practicality of reporting what amount HAS been delivered, in relationship to what was requested. Am I missing something, or maybe I am confused?

Dear denizen2 -

Still I'm afraid data downloading is not as predictable as we'd like it to be.

Example 1 - I have requested ticks for 1 hour back. How does one know the exact number of ticks in a hour before that hour of data downloads and ticks can be counted?

Example 2 - I have downloaded 90% of data and then hit a gap. The 90
% indicator will stay this way for an undetermined period of time, doing nothing.

If I want to make sure data downloading is active, I'd look up the QuoteManager log, it tells in detail what's going on with the datafeed.

Dear denizen2 -

Still I'm afraid data downloading is not as predictable as we'd like it to be.

Example 1 - I have requested ticks for 1 hour back. How does one know the exact number of ticks in a hour before that hour of data downloads and ticks can be counted?

Example 2 - I have downloaded 90% of data and then hit a gap. The 90
% indicator will stay this way for an undetermined period of time, doing nothing.

If I want to make sure data downloading is active, I'd look up the QuoteManager log, it tells in detail what's going on with the datafeed.

"Progress" is just measured in the same units as requested data, and it is either at a stand still (no progress is being made), OR it is increasing (something is happening ).

Example-1: Tick data-bars. You request 1HR worth of 20tick-bars. You say you don't know number of ticks in an hour? Doesn't matter. You are supposed to be 'monitoring' the time-stamps on the tick-data, not the number of bars (of tick-data). So you check the time-stamps and compare with what is required time period back in time.

Example-2: 90% of data has been received (either N-tick or time based bars), as per Example-1, and then the 90% 'progress gauge' stop incrementing? If the resolution of the progress gauge is reasonably small, then even 30 seconds of no progress should be evident as "no change" in the readout. IF that were to happen, then I know it worth my time to 'investigate' further by looking into the QuoteManager log, (as you suggested).

The progress-gauge (say a 4-digit readout, for each symbol) seems to be a reasonable way to inform the user, so I see no reason to 'discount' the suggestions being made in this topic as impractical or not useful . What do you say? Have I finally convinced you that it is useful and important and probably easy (enough) to implement?

It might be possible to calculate the percentage of downloaded time intervals from the requested period, say minutes. It might be possible to use timestamps as you described, but that will be complicated by the necessity to account for session settings and holidays.
In any way, the indicators you're proposing will not indicate true progress of downloading - all they could do is stop when it is interrupted.
If you're concerned about that, why not just watch the QuoteManager log?

Hello Alex,
I was thinking the last two days about the trouble with the datafeeds, and I got an idea of the standing point of view of customers, what do you think?

Disable the data-mode (on-demand) from the chartmodule. And let all this handling do the Quotemanager. So, if customers open the chartmodule they will see quickly the data in the database and then they can decide for each symbol in Quotemanager wether they want, if QM should download missing backfill data or receiving realtime... what ever....
The problem people have, that no data is seen on the chart, but the data are stil in the database (Quotemanager) is that the chartmodule has this different modes: on-demand, ...

A customer opens a new symbol in on-demand mode (chartmodule), because he wants (must load data first, database is empty) an than later all the time he has trouble with charting this symbol...
I dont understand why I should open the chart first in on-demand mode an than close the chart to open it in online or offline mode? This is very circumstantial and unusable if you want download Bid/Ask/Trades in different timeframes for each symbol you want to work with.

I as a customer would prefer to do all the handling (what now is done in the different modes in chart-module) in Quotemanager and do everything what I have to do in the chart-module in offline mode (maybe online mode too).
What do you think? And what other customers think?

I am right that I can not click in Quotemanager a symbol and say: please load for this symbol, Bid/Ask/Trades in different timframes for e.g. 2006.05.24 to 2006.11.24? I am right that I have to do this in the chartmodule?

So, the idea is, have peace in the chartmodule, because it is wilful charting what is in the database.
And the Quotemanager can in the background quietly download backfill, realtime, Bid/Ask/Trades, what-ever for each single or a group of symbols...

Thanks for reply and excuse may english its not my native language.

A Download Scheduler will be added to QuoteManager, doing as you described and also scheduling the downloads for the night hours or whenever you wish. This feature is already in out ToDo list.