Merge of History and Realtime has gaps

Questions about MultiCharts and user contributed studies.
jek
Posts: 166
Joined: 24 Dec 2006
Has thanked: 1 time
Been thanked: 2 times

Merge of History and Realtime has gaps

Postby jek » 10 Oct 2007

When you use ASCII Mapping to add far back history, such as tick data, it is important for the realtime source to fill in from the end of the ASCII data.

As it is, there is a gap from the point that you start running MC.

User avatar
Marina Pashkova
Posts: 2758
Joined: 27 Jul 2007

Postby Marina Pashkova » 11 Oct 2007

Dear Jek,

If the source you use in merging for real-time only supports real-time it can't fill any gaps in the past. The starting point for the data you request form such a source will be the moment of request. It simply can't go back into the past up to the point where your ASCII-mapped data ends. It doesn't provide any historical data whatsoever.

On the other hand, if the data source you are using for real-time supports history as well it could fill the gap going back to the point where the ASCII-mapped data ends. However, in such a case there will be no need in ASCII-mapping since you'll be able to request history directly from your data provider.

jek
Posts: 166
Joined: 24 Dec 2006
Has thanked: 1 time
Been thanked: 2 times

Postby jek » 11 Oct 2007

My data source is TS which does have history.

But tick history stops after 3 months or 6 months.

If you have more history than that, you can augment the analysis.

Your assumption that the data vendor has as much history as the ASCII Merge is incorrect.

Another example is where you want to do some processing on historical information and make the history that is merged be processed in some special way. In this case, you want to merge the historical, perhaps a custom back adjustment, and then use the current prices for the front contract.

Is this more clear?[/quote]

User avatar
Marina Pashkova
Posts: 2758
Joined: 27 Jul 2007

Postby Marina Pashkova » 12 Oct 2007

Dear Jek,

I'm afraid I didn't make myself clear enough.
When you use merging, its very nature presupposes that from the source you designate for historical data only history will be taken. One the other hand, the source you designate as real-time will only give you real time data. That's why it's called real-time.

In your case, to use both the bulk of tick history and real-time quotes for a particular symbol you could do the following:

1. Import ASCII data into the TS symbol (so you'll be using ASCII import and not ASCII mapping). Thus, this symbol will contain all the historical data from the ASCII file.
2. In QuoteManager -> Tools -> DataSources -> TS -> Settings check the box 'Save Data to Local Database'. Thus all the real-time quotes from TS will be saved onto your computer.
3. You will be able to get real-time quotes for the symbol already containing all that historical data. You will also be able to fill the gaps b/w the end of the ASCII data and the beginning of real time.

Let me know if you have any further questions.

jek
Posts: 166
Joined: 24 Dec 2006
Has thanked: 1 time
Been thanked: 2 times

Postby jek » 12 Oct 2007

Thank you for the response.

Yes, I was aware that I could work that way but my experience is that, for a number of reasons, one has to use "reload" on the TS price data and it results in the historical/imported data being lost. One ends up having to re-import it a lot. See other postings in this forum where people describe the need to reload all data from ASCII.

I recall some mention in an earlier post that a data API was being developed. That would help a lot because we could write functions to import/export and check for gaps, fill in gaps, etc etc.

Data management is hard at the moment. This is due to a couple of reasons:

One is mentioned above in that imported data can be thrown away by one's actions and therefore create work. You don't know when this might happen for sure.

Another is that the price database is not high in reliability and visibility leading to people frequently needing to export/archive to ASCII, possibly repair, and then reload.

Another is that one often needs to work with data from multiple sources and compare them.


In some sense, what this suggestion is suggesting is that by filling in gaps between the "history source of data" and the "realtime source of data" (if the realtime source supports it), it becomes possible to create a "golden source of history" that will be insulated from problems in recent times.

This is not necessarily an ASCII mapping issue. It is more of a data management issue.

User avatar
Marina Pashkova
Posts: 2758
Joined: 27 Jul 2007

Postby Marina Pashkova » 12 Oct 2007

Thank you for the response.

Yes, I was aware that I could work that way but my experience is that, for a number of reasons, one has to use "reload" on the TS price data and it results in the historical/imported data being lost. One ends up having to re-import it a lot. See other postings in this forum where people describe the need to reload all data from ASCII.
It is strange that the imported data should be lost after a reload. We have developed this feature in such a way as to ensure that the data in the datebase would not be lost. When the data is requested over a large period, the data that is supplied by the provider is receivd from this provider. The rest is taken from the local data base. No data should be lost.

If you have run into problems with data in the database getting lost after reloads, could you please give us as many details on that as possible?

Thank you.

jek
Posts: 166
Joined: 24 Dec 2006
Has thanked: 1 time
Been thanked: 2 times

Postby jek » 16 Oct 2007

Ok, here are some details.

Every so often I need to clear the TS cache to make sure that stale data is reloaded. When I do this, TS will lose old tick data in TS. I would like MC to retain that old tick data.

Stupidly, I had copied a workspace that had "Merge Data Sources into Single Chart" set and it was asking TS for the history. So of course it didn't see more than 6 months of tick data historically.

When I reset this option and restarted MC, I then saw the longer timeframe history.

Is it required to restart MC when you change the "Merge Data" option?

I tried closing and reopening the workspace but that didn't seem to do it. I had to restart.

However, when I then did a reload just to confirm that reload worked without destroying the history and it did destroy the history and I only had 6 months of tick data again.

So it appears that the meaning of "reload" doesn't just fill in gaps, it loses history data.


I have TS 8.3 build 1419.
I have MC current released version 2.1.999.999.
I have "Save data to local database" selected for TS.

jek
Posts: 166
Joined: 24 Dec 2006
Has thanked: 1 time
Been thanked: 2 times

Changing options in hundreds of workspaces

Postby jek » 16 Oct 2007

To make the change of clearing the "Merge Data" option, I have to laboriously modify more than a hundred workspaces to make the change...

Any suggestions about how to speed up that process?

I hear that workspaces used to be XML files and are not any longer.
If they were accessible/export/import as XML, I could export the XML, change the option and import it.

denizen2
Posts: 125
Joined: 17 Jul 2005

Postby denizen2 » 16 Oct 2007

It is strange that the imported data should be lost after a reload. We have developed this feature in such a way as to ensure that the data in the datebase would not be lost. When the data is requested over a large period, the data that is supplied by the provider is receivd from this provider. The rest is taken from the local data base. No data should be lost.
:? It is ALSO very strange that my two months of intraday tick that has been accumulated for a particular (IQFeed) symbol (@YM#) is NOT being accessed and plotted when I create another NEW chart of the symbol. In otherwords, only the ONE chart that has been used for the last two months (everyday) has accumulated in the local data base. When ever I create a new chart for the same exact symbol and same data-vendor, the amount of history data that is plotted is only what is provided by the data-vendor, in this case 5 days worth of tick data. What happened to all of the past accumulated data?!

My question: IF
(a) the local-data-base has already accumulated a long history of data (say two months worth) AND
(b) the database-keys that are used for storing and accessing this history-data (and real-time data too?) are based on the symbol (and datavendor?),
THEN isn't it correct to assume that any number of charts can LATER be created that plot this same FULL length of data-history (that was accumulated while the first chart plotted that same data)?

To my way of thinking, this issue is very related to the comments in this post topic that indicate a problem of loosing some of the history data under certain conditions. Could it actually be a problem related to HOW *any* new chart (or even new reload??? of a chart?) is supposed to know whether there is any history-data already stored for the requested back-history period?

Could somebody explain what kind of data is stored in the local-database, in terms of what info goes into the 'database-keys' when they are constructed (i.e., symbol + datavendor + resolution + date/times?).

Is the chart ID part of this database key? If so, then that would explain why the new charts can't see the very old data accumulated by another chart :wink: . If that is not the answer, then maybe you could offer another explanation for this lost previously-accumulated-data?


I have encountered this same kind of history-data-issue-problem for more than a year or two, and it never seems to be completely resolved. [btw/ I have the most recent version of MC, in case you need to know].

denizen2
Posts: 125
Joined: 17 Jul 2005

Postby denizen2 » 16 Oct 2007

It might be of interest to know what chart-style-resolution I was using in the post above. My 2 month old chart was plotted as 100 Vol-contracts, which I admit is not exactly the same thing as 100 Tick charts, but that is what was being plotted. So I would expect that I would at least be able to create a new chart plotting 100 Vol-contracts, and ALSO get the whole two months of data that has accumulated in the local storage, right?

I just went back to this original 2-month chart and tried to add a second plot of same symbol, but 1000-Vol-contracts (instead of just the 100 Vol bars). I was expecting that the 1000 Vol-bars would be calculated from the existing 100 Volume-bars, right? But here's what I get: ONLY 5 days worth of bars at 1000 Volume-bars, not the expected 2 months!

Same chart, just added a different resolution of the exact same symbol. So should we expect that the program would always try to download 'fresh' data of a given resolution, regardless of whether that desired history data could be calculated from existing data in the local-database? Could you explain under what cirmcumstances the requested resolution is calculated, rather than downloaded? Is only the raw data of Ticks and 1-minute data ever downloaded, OR does the program sometimes download say 1hr data, if that was the only resolution requested for a given plot, regardless of what resolution is already in the local-database? How are the 'Volume' data-bars created?

This is the kind of technical info that would be great to put into a 'sticky' or a FAQ section, but I am not yet aware of seeing it.

denizen2
Posts: 125
Joined: 17 Jul 2005

Postby denizen2 » 16 Oct 2007

Ugh!!!@%$@%$@# I just lost most of that 2months of data I had been accumulating! How? I thought I would just 'temporily' change the resolution from 100-volume-bars to 200-volume-bars (by use of the symbol formating dialog window). So I got the 200-volume-bars, BUT only the last 4 or 5 days worth! AND then when I tried to switch the resolution back to the original 100-volume-bars..... yep...almost all of the 2 MONTHS of data has vanished! @@^(&*& :evil: :roll:

So much for the 'local-data-base' story keeping my data 'safe' :cry:

Please try to duplicate this same scenario, ... just accumulate some data beyond what is provided by the data-vendor history servers, and IQFeed in particular, since they have (currently) only 4 days of tick data on history servers. Then request some significant amount of history, say 1 month, and record what you actually get plotted. Then a few days later, with the computer continuously running and collecting data, try (a) to load another chart with the same data, and (b) change your existing plot resolution from 100 volume-bars to 200, and then later back to 100. Do you still have plotted ALL of the accumulated data that was supposed to be stored on the local storage? :?: :arrow:

User avatar
Marina Pashkova
Posts: 2758
Joined: 27 Jul 2007

Postby Marina Pashkova » 17 Oct 2007

Hello,

Thanks for drawing our attention to the issue. We are going to check how MC works with those saved data and I will post the results.

Regards.

User avatar
Marina Pashkova
Posts: 2758
Joined: 27 Jul 2007

Postby Marina Pashkova » 18 Oct 2007

Hello everybody,

Thanks for drawing our attention to this reload issue. Our engineers have confirmed that this is a bug: when doing a reload the history that has been accumulated and that goes beyond the scope provided by the data source is not plotted on a chart. However, this data does not get lost. If you close MultiCharts and QuoteManager and then make the request again the accumulated data will be plotted.

We will try to fix this bug as soon as possible.

Thanks again for your help.

denizen2
Posts: 125
Joined: 17 Jul 2005

Postby denizen2 » 18 Oct 2007

Hello everybody,

Thanks for drawing our attention to this reload issue. Our engineers have confirmed that this is a bug: when doing a reload the history that has been accumulated and that goes beyond the scope provided by the data source is not plotted on a chart. However, this data does not get lost. If you close MultiCharts and QuoteManager and then make the request again the accumulated data will be plotted.

We will try to fix this bug as soon as possible.

Thanks again for your help.
Thanks for the bug confirming notice. :D

I and others have pointed out several times that there is a great need for the users to have access to some kind of what might be called a 'data management' facility. If :roll: the users (and yourself) could have easily been able to monitor (and display) some kind of 'database report' of exactly what is, or is not, in the database, then this bug would probably never existed :wink: .

Since the 'database' is such a critical component of the total MultiCharts system, it is a mystery to me why there is not more 'transparency' or 'visibility' of what data is currently in it. Now we only know about the existence of some data (in the database) if it can be plotted. This limitation is VERY frustrating and confusing to everybody, but maybe it is just a 'temporary' situation that is just part of many planned steps to creating a professional-level 'quant' system?

Here's another very powerful 'idea' that is related to the 'database facility' : go even one step further. Please consider (as a future development) the goal of adding a 'DATA-centric' option to the currently existing 'CHART-centric' model.

The idea means that the user would treat the 'database' data as 'global' data, against which the user could write strategy scripts that access multiple charts-based indicators.

It could also be the 'door' for adding a VERY comprehensive facility for
(a) portfolio management , and
(b) a neural-network and fuzzy-network additions.

The current chart-centric approach that is used by almost every charting/analysis application (including MC) is a model that is too restrictive and too in-efficient, because it means that *every* data component to a trading system has to be calculated and stored as data associated with a specific chart, and never can calculated-data be used in other charts. Yes, I know about the GV (Gobal Variable) software that can be used in Multicharts, but that is really just a 'band-aid' that is still chart-centric and it is very complicated to actually use.

You might say that the data-centric-idea is equivalent to adding a real data-base-management system to the front-end of the current charting facility, plus a scripting facility such as EXCEL (but using EasyLanguage, instead of, or in addition to, the BASIC language scripts) that could access any (and all?) calculated indicators (and raw-data too, of course). In a database-centric script, there would be probably some way to limit the 'scope' of variable/data visibility to be global, workspace, or chart. Normal statisical operations could also be performed on any of this data, and results used within any strategy-scripts. 8) , and this would be VERY useful to almost portfolio management scheme, and any trading-strategy that is based on statistical-measuring of the certain market 'conditions'.

I have already posted this 'suggestion' in the suggestioin-topic area, but I have not yet seen any kind of response about it. Obviously, it would be a 'major' addition to the software development goals, but please consider while reaching your current major 'goals'. Why? Because the current goals of adding portfolio management ideas and a neural-network-based facility would be even *more* 'powerful' IF there was ALSO a DATA-centric (and data-visible, and data-base-processing) type model under the whole thing. :idea: Just a 'thought' in my 'dreams' :) . What do you think?

User avatar
Marina Pashkova
Posts: 2758
Joined: 27 Jul 2007

Postby Marina Pashkova » 18 Oct 2007

Hi denizen2,

Thank you for the suggestion. I will get back to you about it after we have discussed it with our engineers. For now just a quick tip on how you can see what's in the database and what's not:
Now we only know about the existence of some data (in the database) if it can be plotted.
To see if you have data for a symbol in the database (and to edit or delete this data) please do the following:

1) Go to QuoteManager
2) Right-click the symbol
3) Choose Edit Data
4) Choose the resolution and the field from drop-down lists
5) Click load.

That will bring up all the data there is in the database for this particular symbol in this particular resolution.

denizen2
Posts: 125
Joined: 17 Jul 2005

Postby denizen2 » 18 Oct 2007

To see if you have data for a symbol in the database (and to edit or delete this data) please do the following:

1) Go to QuoteManager
2) Right-click the symbol
3) Choose Edit Data
4) Choose the resolution and the field from drop-down lists
5) Click load.

That will bring up all the data there is in the database for this particular symbol in this particular resolution..
Thank you Marina, for the above info. I tried it, and it works :) . However, the Edit Data window does not tell you any date/time for the 'beginning' and 'end' of the data, so you must use trial and error, or have some idea of what date you are looking for before you request the 'load' of that data. For example, the Edit Data box labeled as 'Start' Date and Time are filled with some kind of default date/time that does not seem to relate at all to the actual data in the database. So the user would have to 'guess' where is the beginning of the data. :?

So in terms of adding some more functionality to the data-management capability, I would suggest that the *actual* 'Start' date and time, as well as the actual number of records in the database be displayed somewhere in the Edit Data Window.

Another related issue of what is stored in the database: For Tick data, what is the smallest time-increment that can be recorded? The displayed tabular data shows only Hours:Minutes:Seconds, right? But we all know that Tick data can occur maybe more than a 100 times in just ONE second, right?So we need Tick-data precision to be down to at least 10 milleseconds, rather than just seconds.

I have noticed a certain 'artifact' of this lack of high precision in handling the tick data when you look any chart that combines two different time-resolution-bars (for the same symbol). Let me explain:

Say you have a 100-Tick chart that you have added a 1000-Tick-bars (displayed, or not, it doesn't matter). Then you plot two indicators, say some oscillator types, where each indicator is a histogram-type plot, each in its own subgraph region.

Now make one of these histograms have a property of being based on the 100-tick (DATA1), while the other histogram is based on the 1000-tick (DATA2).

You will then notice that there are regular 'holes' or 'gaps' spaces in the 100-based histograms, every 10 bars, i.e., at the exact display location of the vertical-bars in the 1000-bar-based histogram.

Does that mean there is a big hole in the 100-tick data? No, of course not. But it does appear that the display software can not plot two separate indicators at the same exact display-time-location, OR the data being used for display has been rounded down to a lower resolution than the raw data. Which is the case, I don't know. But this 'problem' can cause the user to interpret the displayed data, as being something that it is not, namely 'delayed' a certain small amount. This can be seen as 'significant' by the user, when in fact, it is only a 'artifiact' created by the process of combining the two data series in the same chart.

I would suggest that this issue be addressed first by making sure there is a smaller resolution for recording the time of Tick-data, even if you have to depend somewhat on the 'local' computer-time-base because the data-vendor might not supply the data as having smaller resolution (below 1 second).

Can you (or someone) expand on how this display-artifiact issue is currently being addressed, or will be addressed in the future?

Multi-charts is unique in this capability of plotting different types and intervals of data on the *same* chart, so I know it is not easy to do. I love this capability and use it often, but I have to be VERY careful not to interpret the displayed data incorrectly, mostly because of this 'artifact' described above.

Cheers,
denizen2

User avatar
Marina Pashkova
Posts: 2758
Joined: 27 Jul 2007

Postby Marina Pashkova » 19 Oct 2007

Thank you Marina, for the above info. I tried it, and it works :) . However, the Edit Data window does not tell you any date/time for the 'beginning' and 'end' of the data, so you must use trial and error, or have some idea of what date you are looking for before you request the 'load' of that data. For example, the Edit Data box labeled as 'Start' Date and Time are filled with some kind of default date/time that does not seem to relate at all to the actual data in the database. So the user would have to 'guess' where is the beginning of the data. :?
Dear denizen2,

Why do you think 'start' and 'end' date in the Edit Data window have nothing to do with real data in the database?

Regards.
Regards.

denizen2
Posts: 125
Joined: 17 Jul 2005

Postby denizen2 » 19 Oct 2007

Thank you Marina, for the above info. I tried it, and it works :) . However, the Edit Data window does not tell you any date/time for the 'beginning' and 'end' of the data, so you must use trial and error, or have some idea of what date you are looking for before you request the 'load' of that data. For example, the Edit Data box labeled as 'Start' Date and Time are filled with some kind of default date/time that does not seem to relate at all to the actual data in the database. So the user would have to 'guess' where is the beginning of the data. :?
Dear denizen2,

Why do you think 'start' and 'end' date in the Edit Data window have nothing to do with real data in the database?
Regards.
Hi Marina,
The start date box in the Edit Data Dialog Window contained a date in 2005, not 2007. Clearly a date that is unrelated to what could possibly be in this database.

I was not even using that symbol @YM# before just a few months ago, and did not even have the current IQFeed as my data-vendor. Therefore, I had to conclude that the "Start Data" must be some kind of 'default' date and not related to what was actually in the database.

The 'End date' was correctly stated as the current date, so I did overstate the case on the end of it :oops: . But the Start-date was clearly wrong. I noticed that after manually setting the start date to something about two months ago, then that date was maintained whenever I closed and then restarted the Edit Data dialog. So at least it does 'remember' the last setting. That is good, but only if it is the true beginning date in the database. Or maybe it would be best to separately show the 'actual' beginning date in the database in a separate non-editable box, and then use the 'start' box as either the last-used start date, or if none has been entered by user, then it could show the same as the 'actual' beginning date in the databse. Just a suggestion. :roll:

Are you suggesting that the 'Start' date that is automatically first presented by the Edit Data dialog is intended to be a reading of what is actually the beginning date of the data in the database, for the given symbol and field? It does not appear to be the case, and appears to be a 'bug', or an 'oversight'? It would also be nice to see the total numbe of records in the data base for the selected symbol and field. Is that possible to add?

User avatar
Marina Pashkova
Posts: 2758
Joined: 27 Jul 2007

Postby Marina Pashkova » 19 Oct 2007

Regarding the earlier question about the tick precision:

Both MultiCharts and the data vendours currently used in MultiCharts give ticks with the precision up to the seconds.

The 100/1000 tick bars problem is not related to the precision of those bars. The reason is that in the current solution, it is not possible to completely synchronize tick-bars.

Re edit data: we haven't come across the problem with the wrong start date before. Do you think you might have requested data from IQ Feed going as far back as 2005? Or maybe chose the start date manually?

Regards.

denizen2
Posts: 125
Joined: 17 Jul 2005

Postby denizen2 » 19 Oct 2007

Marina,
RE: Edit Data Start dates

I have 'tested' or 'tried' using the Edit Data now several times more, with different symbols, and in each case the 'start date' that I see displayed looks reasonable, e.g., it is NOT a date in 2005. So I am confused as to why I did see it the first time I did this earlier this week with the @YM# symbol. I did not enter anything 'manually' before it occured, so maybe this is one of those 'mystery' bugs that will never occur again? :roll: I don't don't know, but I will let you know if it ever occurs again.

I would still like to see more database info be more quickly and easily shown to user, e.g., Quote Manager could (should) have a column after each symbol that list the 'Start Date' and another column for Number of Records contained in the local-database.

That way would be so much 'self evident' compared to having to use the Edit Data feature, individually for each symbol. The start date and record number is probably the same regardless of which field is selected, plus it is NOT intuitive for someone to think about Editing the data, when they only want to know if it exist and how far back, right?

This Edit Data feature is certainly useful, BUT it is not usually a good substitue for just reporting the 'status' of the database. At least that would explain why I did not think about it when I have a question of what is in the database :wink: . The suggested approach would provide quick visibility of the whole database, across all listed symbols, while the Edit Data window... well you have to select... and wait a long time to load... and then you finally get to 'edit' something, when you really only want to know if it even exist. As they say, 'the wrong tool for the job'?

Sorry about running on so long about this... but it seem that I have 'harped' on this issue now several times, and no body there seems to appreciate the reasons for making the database actually look like a *database* to the user, instead of looking (to the user) like maybe just a text file :wink: editor.

So thanks for the help... at least I now understand there is a Edit 'feature' that I can use in a pinch when I really want to find out if there is any data in the database back to a given date.

Cheers,
denizen2

User avatar
Marina Pashkova
Posts: 2758
Joined: 27 Jul 2007

Postby Marina Pashkova » 22 Oct 2007

Hello denizen2,

To tell the truth, we aren't planning to add the features you are talking about (an extra column for the start date and the number of records) in the nearest future. But we are going to consider implementing them in the future MC versions.

Regards.


Return to “MultiCharts”