Data Management BUG? More Evidence!

Questions about MultiCharts and user contributed studies.
denizen2
Posts: 125
Joined: 17 Jul 2005
Has thanked: 8 times
Been thanked: 1 time

Data Management BUG? More Evidence!

Postby denizen2 » 24 Feb 2007

There is one major frustrating 'issue' that never seems to go completely away, even with the newest beta 2.0.777.777. I have been a beta tester since before version 1.0. Even though there have been VERY many 'bug' issues resolved, I have not *yet* been able to escape seeing some kind of problem with the data management that usually manifest itself as 'missing data', or data that 'does not load', or data that loads 'too slowly'.

The latest episode of this saga is happening today, Saturday, Feb 24, 2007, when the Forex markets are totally 'closed'.

My 'problem': I can not seem to load a new chart with 100 days of 100-tick bars from eSignal, even though there already exists on my computer some 'old' charts with this *same* symbol, resolution, and date range that I am requesting today (in the NEW chart).

The glaring contradiction here, of coarse, is this: IF there is already data loaded in some chart, then that would imply that the data MUST also have been stored in the DATABASE, right? So regardless of whether the eSignal (history OR real-time) Tick-servers are available or not, there is (or should be) history data in my local database, right? Granted, the data stored in the database is ONLY the smallest resolution type data, i.e., the 'Tick','Minute', and/or 'Day', as selected in 'Fields to Collect'. BUT, if there is 100-tick data for a given symbol in a chart for the last 60 days (up to the current last trading day, Friday, 2/23/2007), THEN there should be at least the 1-tick data in the database for that same period of dates, right? However, in my case we see the 'status' line showing "Establishing connection...", and then NOTHING else happens...! Apparently, the MultiCharts data management software is trying to access ONLY the eSignal data-server(s), which are not available on Saturday, right? The answer to why this is so may also happen to 'resolve' some of the other complaints about 'missing' data?

I have attached a zip file of the related tsserver.exe log file. My questions would be:

(1)IF indeed the data being requested is ALREADY in the database, THEN WHY is it necessary to establish *ANY* connection, if by 'connection' it is meant 'internet connection' to the *remote* eSignal server? Perhaps what is being reported is not exactly what it appears to be? Is there any way to distinguish (or observe) whether the chart-data is being requested from the remote data sources, OR the local database? [More definitive status-reporting regarding whether local or remote data is being accessed would also be very helpful to everyone concerned.]

(2) I notice that the three database files, tsstorage.gdb, fbportfolio.gdb, and tscache.gdb, all have a 'Date Modified' stamp of 2/20/2007. This computer has been running and continuously updating the charts all this last week until now, Saturday, 2/24/2007, so why is the time-date stamp not 'modified' to show the last date modified, which is presumably yesterday, Friday, 2/23/2007? Does this evidence imply that the database has not actually been updated all week and my existing charts have ALL been loaded with ONLY the REAL-TIME and HISTORY *downloaded* data from esignal??? I would expect that the database should contain all of the history data that was used in existing charts, right? And if so, THEN the database should have a date stamp of the last received history update, right? But that is not what I am seeing!

As I am preparing this bug report, I needed to zip the log files that are attached. However, the zip program complained that I would have to exit the programs that already have the log files open. So I closed down the whole MultiCharts application, and then created the zip version of the log files.

When I restarted MultiCharts and re-loaded the same workspace and the (new) chart that wouldn't load before this (as already explained at length). Wha..... NOW the chart loads very quickly with those 'missing' 100-tick databars, AND the database date-stamps are now 'updated' to the current date and time :>))!

So did it load all of that 100-tick data from the *local*-database? Or did all of the 'history' data come directly from eSignal's 'history' data-servers (on this Saturday, during closed market hours)? The log files were zipped BEFORE re-launching MC, so they show only what was being attempted in the state before re-launching [see file 'TSSupplier-log-1.zip' ]. The MC software was then shutdown a second time, and another zip file of the log file was created, and named 'eSignal[tsServer]-log-2.zip'. Maybe, the answers will be evident by comparing those two log files?

Maybe this being a Saturday is very significant in sheding some light on some kind of QuoteManager or tsserver.exe program logic that is amiss? On Saturdays, the eSignal real-time and history tick-servers can not respond, right? But the fixed-time interval servers seem to work ok? [I was able to load 100-second-bars, but NOT 100-tick-bars, so problem seems to only involve tick-bars]. Maybe your staff needs to do some more testing during the Saturday times?

All of us users just want to see the data-management issues to be resolved so we can work 24/7 without these frustrations of missing data or data that won't load. We all understand that there will be times that the data-vendor is the 'problem', and not the MC software, but we have so little information even about what exactly 'should' be expected on these data-loading operational issues that makes it very difficult for us to help you 'debug' something like this. Maybe it would be helpful to everyone if someone could spell out exactly when and how data is being obtained from the local-database vs the remote-internet accessed data-vendor's servers. AND then make the status line report more explictly which data source is currently being accessed. Is that possible to show us on the 'next' beta?
Attachments
TSSupplier-logs-1.zip
Log-1
(76.64 KiB) Downloaded 262 times
eSignal[tsServer]-log-2.zip
log-2
(1.05 KiB) Downloaded 266 times

denizen2
Posts: 125
Joined: 17 Jul 2005
Has thanked: 8 times
Been thanked: 1 time

Postby denizen2 » 27 Feb 2007

Almost three days have passed since I posted the above message, yet there does not seem to be any 'response' :roll: . I know it is a very long message to read, but it reports my own investigation that seems to confirm there is a 'bug' that is very important to 'fix'. Specifically, how the MC software determines what data needs to be requested from the remote data vendor vs what data is already in the local data storage.

Just because I was able to finally 'get' the data loaded in the chart by quiting the MC program and re-launching should also tell you something, plus it is maybe not just a random thing that the problem occurerred when the eSignal FX markets were closed.

It would be appreciated if someone in TSSupport would please respond to at least some of my questions and the information provided. :P

denizen2
Posts: 125
Joined: 17 Jul 2005
Has thanked: 8 times
Been thanked: 1 time

Postby denizen2 » 28 Feb 2007

Here's some more evidence of some kind of data managment 'bug': [This is different than the one presented in my previous post above]

The attached screen shot (of two monitors) shows missing six hours of 5-min bars beginning around midnight 2/26/2007, for EUR A0-FX symbol (eSignal,... see left panel, middle sub-graph). Note: The window format has been set to show 'weekend' or 'empty' data-times too.

Also look at the right side of screen you see the eSignal-Chart of the same symbol and 5-min resolution bars.... and you see the same gap for the empty weekend, BUT there is NO gap during the 2/26/2007 24-hour period. That tells me that the servers at esignal are working ok for that data period, right?

So why does the MultiCharts NOT print any 5-min bars for the beginning hours on Monday the 26th? Actually, this is screen shot is the 2nd time I re-loaded the chart, because the first time the missing block of bars was just after the beginning of Wed, 2/28/2007, and now that period looks ok, but it has shifted to the 26th! How could this happen?

Please note that these problems have all occured when I have combined tick-bar and time-interval bars on the same chart. In this case the tick-bars are 100-tick bars. I have also included a third data series, 150-seconds per bar, and it looks ok, but the 5 min-bars has this approximately 6 hours of missing data?

Can somebody explain to me what data is being stored as 'history' data? Is is just the raw tick-data, and the raw 1-minute data (plus Daily data bars)? Then how is the 150 second-bar data created? From the tick-data? And how is the 5-min bars created? From the 1min-data, or the tick-data, or what?

Since the missing data seems to be at the beginning of the local-24hr day, is there some possible problem related to how the 'session' is defined when the local-time is different than GMT zone?

Does anybody else have any similar screen shots to show TSSupport? If my memory serves me correctly, there have been significant number of forum post of similar problems.
Attachments
missing data.GIF
MC & eSignal Charts compared
(206.71 KiB) Downloaded 1187 times

User avatar
Andrew Kirillov
Posts: 1589
Joined: 28 Jul 2005
Has thanked: 2 times
Been thanked: 31 times
Contact:

Postby Andrew Kirillov » 01 Mar 2007

Dear denizen,

We are analyzing your issues. You will receive a detailed answer soon.

denizen2
Posts: 125
Joined: 17 Jul 2005
Has thanked: 8 times
Been thanked: 1 time

Postby denizen2 » 01 Mar 2007

Dear denizen,

We are analyzing your issues. You will receive a detailed answer soon.
Thank you Andrew. Perhaps you have enough data already to look at, but here is something more I discovered this morning.

If you look at the attached images, the first one has two 5min dataseries. The top one is the Bid, and the bottom one is the Ask prices for EUR A0-FX (eSignal). Both data series look ok (i.e., no data gaps).

The next thing I did was just modify the upper chart to change from 5MIN to 100-TICK bars. The result was that the upper chart displayed the 100-tick bars ok, BUT look what happened to the lower chart of 5MIN bars! I had NOT even touched that data series, but AFTER changing the TOP data series FROM a fixed-time-interval TO a tick-based chart, THEN the other time-interval chart (5min-Ask) was 'broken' into a gap that began at the beginning of the 24HR (local-time) period.

Maybe somebody could try to repeat the same two-step experiment?
Attachments
5min bid ask data-no problem.png
5MIN Bid & Ask-No problem
(377.92 KiB) Downloaded 1191 times
100tick bid -5min ask data-NOW problem.png
Change to 100tick-bars & see other 5min bars now have gap!
(344.95 KiB) Downloaded 1193 times

User avatar
Kate
Posts: 758
Joined: 08 Dec 2006

Postby Kate » 07 Mar 2007

Dear Denizen2,

Please review this topic: http://forum.tssupport.com/viewtopic.php?t=3283

denizen2
Posts: 125
Joined: 17 Jul 2005
Has thanked: 8 times
Been thanked: 1 time

Postby denizen2 » 07 Mar 2007

Dear Denizen2,

Please review this topic: http://forum.tssupport.com/viewtopic.php?t=3283
Thank you Kate. I have 'installed' the 'hotfix' files as suggested in the above topic, and the experiment-test was repeated [i.e., changing one sub-graph from 5min to 100tick bars, described in previous post]. The problem previously reported did NOT occur, so looks like the hotfix worked. Thank you, good work!

(1) But I am very confused about why the eSignal data-access components are even involved. Maybe you could explain more what kind of processing is happening when I changed the top-sub-chart of 5min bars to 100tick bars, and WHY the EXISTING 5min bars in the OTHER sub-chart should suddenly display with a large gap where there was already data?

(2) Here is another puzzling issue that is probably related: Consider the following exercise: I have two charts with the SAME symbols and same data-resolution, and even in the SAME workspace, and THEN close (with save workspace), and THEN almost immediately re-open that workspace.

I would expect that just re-opening a workspace of two duplicate charts would NOT involve downloading that data (from eSignal) again, right? And IF for some reason more data (maybe recent 2 minutes of data?) is needed, then it should NOT be downloadaed TWICE, right?

(3) While you are explaining what processing we should be expecting in the above scenario, could you also explain the meaning of the 'status' line in each of the two charts being loaded? These status lines typically say 'xxxx quotes received' where the 'xxxx' number is continously being incremented, until it finally says 'connected'.

The choice of words used in these status readouts seems to imply that the data is being downloaded (again) from eSignal, AND that each chart (with same exact data) is being separately downloaded because these status lines are completely different for each. So should I take the status meaning 'literally' reported, or maybe there is something more subtle that I am missing?

(4) What role is the data-cache play in all of this? Isn't the data-cache supposed to hold all of the previously downloaded (history+real-time) data? If so, then why do we need to see status in each chart as 'xxxxxxxx quotes received'? i.e., Quotes from eSignal, OR quotes from the data-cache?

I would expect to see ONLY a small number of quotes received that represents ONLY the real-time data that was missed during the short duration between saving and reloading the chart, right? Please explain what we should be 'expecting' in this re-load-a-workspace type scenario.

Perhaps all of these questions have already been answered or discussed, but I have not yet seen them :>)).

'Beta Testing' is so much 'fun' for everyone when we have an understanding of what is a 'feature' and what is still a problematic 'feature-in-progress', right?
:)
Cheers,
Denizen2

User avatar
Andrew Kirillov
Posts: 1589
Joined: 28 Jul 2005
Has thanked: 2 times
Been thanked: 31 times
Contact:

Postby Andrew Kirillov » 14 Mar 2007

And if so, THEN the database should have a date stamp of the last received history update, right?
No they should not. Database files' timestamps are updated only at the MultiCharts shutdown.

denizen2
Posts: 125
Joined: 17 Jul 2005
Has thanked: 8 times
Been thanked: 1 time

Postby denizen2 » 14 Mar 2007


I would expect that just re-opening a workspace of two duplicate charts would NOT involve downloading that data (from eSignal) again, right? And IF for some reason more data (maybe recent 2 minutes of data?) is needed, then it should NOT be downloadaed TWICE, right?

(3) While you are explaining what processing we should be expecting in the above scenario, could you also explain the meaning of the 'status' line in each of the two charts being loaded? These status lines typically say 'xxxx quotes received' where the 'xxxx' number is continously being incremented, until it finally says 'connected'.

The choice of words used in these status readouts seems to imply that the data is being downloaded (again) from eSignal, AND that each chart (with same exact data) is being separately downloaded because these status lines are completely different for each. So should I take the status meaning 'literally' reported, or maybe there is something more subtle that I am missing?

(4) What role is the data-cache play in all of this? Isn't the data-cache supposed to hold all of the previously downloaded (history+real-time) data? If so, then why do we need to see status in each chart as 'xxxxxxxx quotes received'? i.e., Quotes from eSignal, OR quotes from the data-cache?

I would expect to see ONLY a small number of quotes received that represents ONLY the real-time data that was missed during the short duration between saving and reloading the chart, right? Please explain what we should be 'expecting' in this re-load-a-workspace type scenario.

Perhaps all of these questions have already been answered or discussed, but I have not yet seen them :>)).
I know you guys are VERY busy.... but it would be *very* useful to everyone, in my opinion :) , if someone could please respond to the questions/issues cited (again) above. Why is this important to us beta testers? Because if there is no informational statements available (to us) about the 'intended' operational flow of data, how can we most accurately report any 'defects' in that intended functionality?

There is one more specific suggestion I have that would 'improve' the two-way communication between the testers and the development staff on the subject of ongoing data-management related bugs: Please enhance the chart-status-line to include something about WHERE the data is being REQUESTED from , and then as arriving, where it is being LOADED/DOWNLOADED from, i.e., in simple English it is either (a) 'local' data-cache, OR (b) 'remote' data-servers name, e.g., 'RT-eSignal', 'Histo-eSignal'.

As it is now, the user is shown a status line that says something like "xxxx quotes received".... and then later "connected". But this does not tell us WHERE the data is coming FROM. There really needs to be more 'transparency' on where exactly the 'request' for more data is being sent TO, and being received FROM. The current status-line reporting does NOT tell us enough info to make any assessments about whether an observed 'data-gap', or an observed 'data-request' is (a) an anomaly, or (b) normal-expected behavior.

In addition to enhancing the status-line to include 'source' info, I would also suggest that on the 'View' menu a command be added to 'Refresh' the display (meaning refresh from the local-cache-ONLY, i.e., NO external history nor any real-time data is requested). We now have the command for 'Reload', which is very good to have, but maybe we just want to VISUALLY 'confirm' exactly what data is now in the local-cache.

Under the topic of 'explaining' the intended data-flow functions when any data-symbol is loaded, or re-loaded: Could you confirm whether ALL data-bars displayed have to be calculated on the fly from the smallest resolution-data, i.e., tick-data, 1min-data, 30-sec data, DAILY data, or what? So does that mean, for example, that 15 minute bars of the same symbol on the same or different charts (in the same workspace) will ALWAYS be re-calculated EACH and EVERY time that SAME data is loaded or re-loaded? If so, then that would explain why there is so much delay in displaying (or 'downloading'?) the data, right? That is another reason why it is 'better' to show the user more info on what is happening so the user is not confused into 'assuming' that data-quotes are being 'downloaded', when in fact they are just being RE-calculated from the local-store :wink: , right?

Thank you for listening to my suggestions of how to improve the whole beta-testing experience for everyone, as well as everybody's best efforts to contribute to a great software product. :)

Cheers,
Denizen2

User avatar
Andrew Kirillov
Posts: 1589
Joined: 28 Jul 2005
Has thanked: 2 times
Been thanked: 31 times
Contact:

Postby Andrew Kirillov » 15 Mar 2007

I would expect that just re-opening a workspace of two duplicate charts would NOT involve downloading that data (from eSignal) again, right? And IF for some reason more data (maybe recent 2 minutes of data?) is needed, then it should NOT be downloadaed TWICE, right?
If you already have N-tick chart, data for the same second will be taken form RAM. If you have N-minute chart data for the same second chart will be taken form the storage and the last data (for instance, 2 minutes) will be downloaded form the servers.
While you are explaining what processing we should be expecting in the above scenario, could you also explain the meaning of the 'status' line in each of the two charts being loaded? These status lines typically say 'xxxx quotes received' where the 'xxxx' number is continously being incremented, until it finally says 'connected'.
In case there are several charts in the workspaces that are receieving data, those charts that have already received all requestedd data and are ready to be plotted will have "Connected" in the status line untill all other charts receive all the necessary data. And only after that all the charts will be plotted.
The choice of words used in these status readouts seems to imply..
So far MultiCharts doesn't know where data is taken and that's why this status line will remain.
So should I take the status meaning 'literally' reported, or maybe there is something more subtle that I am missing?
Please see the above explanation.
What role is the data-cache play in all of this?
Isn't the data-cache supposed to hold all of the previously downloaded (history+real-time) data?
When you are plotting N-tick charts they are created from raw tick data and MultiCharts has to do with a tremendous amount of data. The cache can considerably speed up loading of N-tick charts if such chart has already been created.
If so, then why do we need to see status in each chart as 'xxxxxxxx quotes received'? i.e., Quotes from eSignal, OR quotes from the data-cache?
MultiCharts doesn't know where data is taken from.
I would expect to see ONLY a small number of quotes received that represents ONLY the real-time data that was missed during the short duration between saving and reloading the chart, right?
Please explain what we should be 'expecting' in this re-load-a-workspace type scenario.
In reality only missing data is downloaded from the servers, all the rest is taken from the storage. But since MultiCharts doesn't know where data is taken the total number of quotes is displayed in the status line.
Why is this important to us beta testers? Because if there is no informational statements available (to us) about the 'intended' operational flow of data, how can we most accurately report any 'defects' in that intended functionality?
You could pay attention to the loading speed: data is always loaded faster from the database than form the Internet.
Please enhance the chart-status-line to include something about:
i.e., in simple English it is either
(a) 'local' data-cache,
OR
(b) 'remote' data-servers name,
e.g., 'RT-eSignal', 'Histo-eSignal'.
Not all users want to receive extra information in the status line. To get the information where quotes are taken either form the database or form the Internet please take a look at QuoteManager logs. If data is downloaded form the internet you will see many data requests.
We now have the command for 'Reload', which is very good to have, but maybe we just want to VISUALLY 'confirm' exactly what data is now in the local-cache.
Reload is working as follows: MultiCharts compares data stored in the database and on the datafeed's servers. The data that is present in MultiCharts but is missing with the datafeed remains unchanged. The data that is present with MultiCharts and the datafeed will be overwitten with the datafeed's data.
Could you confirm whether ALL data-bars displayed have to be calculated on the fly from the smallest resolution-data, i.e., tick-data, 1min-data, 30-sec data, DAILY data, or what?
N-ticks and N-seconds are created from tick data, N-minutes are created from minute data, N-days are created from daily data.
So does that mean, for example, that 15 minute bars of the same symbol on the same or different charts (in the same workspace) will ALWAYS be re-calculated EACH and EVERY time that SAME data is loaded or re-loaded? If so, then that would explain why there is so much delay in displaying (or 'downloading'?) the data, right?
Yes, but this process is lightning fast and does not cause delay. Judging from your workspaces, you are requesting hundreds of thousands of ticks and it causes the delay since the program has to process such amount of data.

denizen2
Posts: 125
Joined: 17 Jul 2005
Has thanked: 8 times
Been thanked: 1 time

Postby denizen2 » 16 Mar 2007

Thank you Andrew for your long reply, and responding to most of my queries :D .

You say that most users do NOT "want to receive extra information in the status line. To get the information about where quotes are taken, either form the database or form the Internet, please take a look at QuoteManager logs. If data is downloaded from the internet you will see many data requests."

I'm sorry to tell you this, but I would suggest just the opposite is true :roll: ! Users would *much* rather see a status-line that identifies the SOURCE of the data in a *simple* English phrase, e.g., "RealTime-remote-eSignal-xxxx quotes", OR Historical-local-cache-xxxquotes", rather than the fleeting and cryptic stuff that is available in the QuoteManager 'Log". Trust me :P !! That 'log file' is useless to anyone except the developers who are 'investigating' a QuoteManager issue/bug, after the fact.

It is far too difficult to even read this log file until AFTER it has stopped scrolling, plus it does not even say whether ANY data is coming from the local-cache.... all you see is how many quotes are requested and when the request is 'finished'. Plus, even more confusing, .... the data in the 'log file' pertains to ALL charts, and ALL 'active/online' data request, so the one symbol that I might be interested is buried among a lot of other data request that have no relevance to my query, in a particular chart.... So I hope you get the picture from the standpoint of the user by now?

From your response, I now understand why the data-source (local or remote) is not presented in the MC Chart window.... you say that this information is not even passed from the QuoteManager module to the MultiChart charting module. So this is the reason the status line just reports "xxxx quotes" received..... mmmm :oops: ?

I have to ask: Is the lack of this kind of status info from the QuoteManager (just maybe) the reason there are 'issues' with the data-management functions, such as already in evidence in this forum topic? :roll: ? :? How many times have you had to field questions from users that are confused about whether a problem is related to the data-vendor, OR the MC/QuoteManager software? How many issues could have been spotted more quickly by just having clear and specific info that validates or in-validates any discrepancies between what is requested and what is actually available data, at any given time? How many 'inconsistent' but 'irritating' hang-ups do we want to endure during the 'beta testing' stage, before we call it 'stable' and ready for 'production use'?

In any case, I will state that this user would much prefer more 'transparency' , rather than more 'hiding' of information concerning a particular chart and a particular action for loading/updating a particular data-series.

It might even be determined (by further investigation) that NOT having a 'better protocol' for communicating between the two modules has been the source of many 'bugs' , or at a minimum, the source of much confusion (to users AND developers) about where/why data-chart-loading-delays are sometimes encountered. I will assure you, there is NOTHING more 'irritating' to the users than 'being left in the dark waiting for something to happen', right?

So I hope my 'soap box' approach to this topic has gotten the attention of those making decisions on 'priorities' for the developers. The 'stability' of this software will probably be 'judged' mostly on such issues as what I will call 'graceful OR ungraceful error-handling'. I think we're 'almost there'. Perhaps the next version of this beta?

BTW: I understand that no software is perfect, but its imperfections should be as transparent as possible, otherwise there is little chance to 'fix' the errors (in any reasonable time-frame) . The bug-ticket system referred to by the previous 'guest' post sounds like a *very* good idea to help both the beta testers and the developers focus most effectively on achieving the goals and features we all are striving for.

Thanks again, Andrew, for answering my questions. Your response greatly supports the process here in this forum for identifying the issues that might need more attention. Maybe you would like to know that I'm happily and thankfully using this product *everyday* (including the weekends too!), and depend on it for ALL of my charting/analysis/trading work, AND I much prefer this software to the TS software,.... that I used to use, but have now canceled all accounts there. :D

Cheers,
denizen2


Return to “MultiCharts”