Reload issue

Questions about MultiCharts and user contributed studies.
2haerim
Posts: 502
Joined: 01 Sep 2006
Been thanked: 2 times

Reload issue

Postby 2haerim » 15 Apr 2008

The scenario below applies to all kinds of charts but for simplicity I would use daily example scenario.

[1] QM currently has daily data from 2000/1/1 to 2008/4/15
[2] MC plots a daily chart with the range of 2000 days back and it plots 2000 bars
[3] I press Ctrl+R to reload and MC requests 2000 daily data from the datafeed server.
[4] Now there can be two cases in the datafeed server side:
1) The datafeed server has more than 2000 daily bar data
=> This is just fine and MC plots 2000 bars again using the newly downloaded daily data
2) The datafeed server has 1000 daily bar data only
=> This time MC plots only 1000 daily bars even though QM has all the data for the next 1000 daily bars


In case 2) above, MC should be able to plot 2000 daily bars using both the newly downloaded 1000 daily bar data and existing 1000 daily bar data stored already in GDB.

User avatar
Marina Pashkova
Posts: 2758
Joined: 27 Jul 2007

Postby Marina Pashkova » 15 Apr 2008

Hi HaeRim,

Below, please see an excerpt from our help file regarding the reload. If in your case it works differently, please contact us via LiveChat.

"Reloading will redraw the chart based on a fresh set of data from the data source. In the process of reloading, a new set of data for the entire range of the chart is downloaded from the data feed. Any existing data, for which no replacement data is available from the data source, is retained."

2haerim
Posts: 502
Joined: 01 Sep 2006
Been thanked: 2 times

QM does NOT ALWAYS retain existing data ...

Postby 2haerim » 15 Apr 2008

Marina,
Reloading will redraw the chart based on a fresh set of data from the data source. In the process of reloading, a new set of data for the entire range of the chart is downloaded from the data feed. Any existing data, for which no replacement data is available from the data source, is retained.
You are wrong in that the existing data is retained in QM if the new set of downloaded data is somehow less than the existing data before download. QM seems to first of all delete the existing data and then wait to receive the new set of data from the data source. However, depending on the data vendor's policy for keeping old history data, the amount of newly downloaded set could be 1) always full (this means the data vendor keeps infinite amount of history data and practically no data vendor does so as far as I know) 2) less than it was before because data vendor had deleted old data according to their policy. So there could be a full data set available yesterday, but suddenly today some old history data is not available any more. Therefore QM should check if the newly downloaded data set is full or not by comparing the timestamp of existing data and newly downloaded data. If the newly downloaded set is less than the existing data, then the existing data range not downloaded from the data source shoud NOT be deleted so that MC can plot the chart using both existing and newly downloaed data. That is, QM should check if the newly downloaded data covers the full requested range and determine if it should delete whole existing data or only part of existing data corresponding to the range of newly downloaded data.

This reload issue applies to all the datafeed. You may try yourself using GS feed as follows:

[1] Run GlobalServer itself in offline mode so that the tsGSDataFeed keeps the same data not receiving additional data during this experiment.
[2] Next, delete a symbol X and insert it back again to completly delete data for the symbol X including tick/min/daily.
[3] Now, import daily data for X into GlobalServer for the range of 2000/1/1 to 2008/4/15 and then GlobalServer itself will have more than 2000 daily data rows. In my case 2040 daily data were available. Be careful not to import other resolutions(tick/min) at all.
[4] Run QM and delete the symbol corresponding to GlobalServer's X and clear cache if necessary. Now, no symbol and no data for X in QM.
[5] Run MC and type "GS:X" from the command line to plot the symbol X for daily bar chart for the range of 2000 Bars back. The chart should plot exactly 2000 daily bars(Surprisingly, it plotted only 1991 daily bars, but this is another issue to be posted in different thread in this forum and possibly 9 missing bars are for some holidays.). In my case it plotted from 2000/04/06(BarNumber=1) to 2008/04/15((BarNumber=1991).
[5] Check to see QM has saved 1991 daily data. In my case it did have 1991 daily rows. In result, QM has 1991 daily dara rows and GlobalServer have 2040 daily data rows. So far so good.
[5] To test the reload issue I raised, please delete the daily data for X from the GlobalServer leaving only the last 1000 daily data rows. In my case I had to delete from 2000/1/1 to 2004/04/03. Now, QM has 1991 and GS has 1000 daily data rows, respectively. And of course, MC shows 1991 daily bars at this moment.
[6] Now time to reload. Press Ctrl+R to see what MC plots. It plots 1000 daily bars only. And the 991 bars have gone from both MC and QM.
=> QM should have checked if the newly downloaded data set has all the data for the requested range and if the new data set is less than the existing data set, then it should have not deleted the 991 daily data rows since the newly downloaded set of data has only data for the last 1000 daily bars. This could happen when the data vendor's policy is to keep limited range of history data for practical reasons. QM should only overwrite for the range of newly downloaded set of data after checking the timestamp of the newly downloaded data.


I think this reloading issue can be solved if QM does not delete the existing data but just overwrite it with newly downloaded data by checking the timestamp of each data row.

I am quite sure that I experimented a lot in many ways and described this issue in very clear way (spending more than 6 hours) to be reproducible, so please try this in your side.

User avatar
Marina Pashkova
Posts: 2758
Joined: 27 Jul 2007

Postby Marina Pashkova » 17 Apr 2008

HaeRim,

The test with Global Server cannot be considered a valid reload test because QuoteManageer does not save any data from Global Server at all.

Regards.

2haerim
Posts: 502
Joined: 01 Sep 2006
Been thanked: 2 times

Again, I reproduced the reloading problem with IB

Postby 2haerim » 28 Apr 2008

Marina,

In the initial post I described the reloading issue with GS datafeed. It is simply reproducible by changing GS datafeed plugin to save data into QM.
I used the original regular GS datafeed plugin with only CanSaveFromStrorage option changed from FALSE to TRUE.

Since your engineers keep saying that there is no such problem, I've decided to reproduce this reloading bug with IB and here it goes as follows:

[1] Run TWS and add a future symbol KSM8 (its root is K200).
[2] Run QM, select IB and add the symbol KSM8 into symbol portfolio.
[3] Run MC and plot IB's KSM8 with Daily resolution, 3000 Days back.
[4] IB does not have all the 3000 daily data for KSM8 and therefore MC will plot only from 2007/06/15(Barnumber=1) to 2008/04/29(Barnumber=141). The # of downloaded bars are 140. Check this number from QM's Edit Data and also Barnumber from MC .
[6] Close the KSM8 window.
[5] Import the attached daily data into QM's KSM8. The data ranges from 1996/05/05 to 2007/10/12. The number of daily bars in this file is 2800.
[6] After importing, check again the total # of data rows saved in QM by opening Edit Data window and pressing Load button for daily. You will see 2905 daily rows. This number is 2800+140-overlapped daily data(35).
[7] Open KSM8 chart again with the same setting (daily, 3000 days back).
This time you will see the chart plotting from 1996/05/03(Barnumber=1) to 2008/04/29(Barnumber=2906). Now QM and MC both have correct # of bars.
[8] Finally, press Ctrl+R to reload the whole chart.
This time you will see the chart for the range from 2007/06/15(Barnumber=1) to 2008/04/29(Barnumber=141). All the bars from 1996/05/03 to 2007/06/14 are gone. Check again QM's Edit data to find out the total # of saved daily rows is 140. It was 2905 before reload.. The others are all deleted.

This is the solid proof of MC/QM reloading bug, isn't it?

This happens because QM delete all the data for reload range before downloading the data from IB. Instead QM SHOULD wait till all the newly requested data is received. After then QM should check if it covers the required reload range. If so, then it's ok to delete the existing data from QM and save the newly downloaded data. In this case MC will plot the whole range again using the newly downloaded data.
If not, however, QM should NEVER delete the existing data for the whole reload range but only for the range of newly downloaded data. In the above test, QM should have NOT deleted existing daily data from 1996/05/05 to 2007/06/14 since IB only sent data from 2007/06/15 to 2008/04/29. This time MC should plot the whole chart both the existing data and newly downloaded data but QM already deleted them.

It is too much for me to prove this bug repeatedly and I don't know how else I can prove my assertion. I've verified this for GS datafeed, IB datafeed and my own datafeed. If I can have temporary access to any other data source, I can prove this bug again.

Please fix this bug because this bug is very serious in that once traders press Ctrl+R to reload, signals will definitely get corrupted and wrong unwanted signals could be sent or wanted signals would not be sent without knowing.
Attachments
10199000_raw_960503_071012_Day.txt
(228.45 KiB) Downloaded 312 times

User avatar
Marina Pashkova
Posts: 2758
Joined: 27 Jul 2007

Postby Marina Pashkova » 08 May 2008

We are looking into this HaeRim. I'll post the test results here.

User avatar
Marina Pashkova
Posts: 2758
Joined: 27 Jul 2007

Postby Marina Pashkova » 13 May 2008

Hi HaeRim,

The problem you are describing above has been confirmed - for daily charts from IB. Reload for minute charts works the way it's supposed to.

The bug will be fixed in future MC versions.

Thank you for reporting this bug.

Regards.


Return to “MultiCharts”