How do you manage you quote data base?  [SOLVED]

Hi,

I get realtime CQG-data through AMP and they advertise also historical data.
For my symbols of interest I collect data related to ask, bid and trade hoping to have all in the correct sequential order.

There are several reason why my charts a plotting gaps, as there is missing data.
I thought "no problem you could use the reload function of MC".
I expected that only the gaps are filled then and pushed the button.

As a surprise more data than expected was pulled from the historical servers.
I started to be concerned how this is impacting my statistical analysis I had conducted lately and checked again.

In deed I found that for several days some things changed after the reload: Open, High, Low, Close, Volume, range. But additionally for some days where my statistical analysis says I should have data - thus no gap - I now have gaps after the reload.

My English is far from perfect so I read this several times but still are not sure if I understand correctly:

https://www.multicharts.com/trading-sof ... ing_Charts

My concerns are:

1. If I have already data in my data base with specific timestamp of the times I received the real time data and the reload source actually has the same prices/volume but slightly different timestamps (when they received data or what is transmitted by exchange) and MC would keep my data but only load data for other timestamps, than I would have to much data points in my data base.

2. If simply MC reloads all the data from the data source, that means kick out the data I collected so far, and duplicated the data from the data source, than I would be completly dependant of the quality of the data source....with all its errors.

3. I read now several times that ask/bid data in external data source are not maintained as properly as trade data. And even if they have complete data sets that bid, ask and trade are not necessarily in sync if you pull historical data. Thus this two points would also mess up a lot of things.

So I am very interested in your points of view.
Which other problems are there I do not see at the moment?
How do you handle all the problems and get a reliable and complete data base?

Thanks and best regards
asyx

For statistical work you should use external datafeeds, for instance, IQFeed, eSignal...
Broker feeds are mainly for execution purposes, not for statistical analysis, specially historical analysis.

Every time you reload a MC chart all data is downloaded again from the feed servers (ie, all data that was loaded on the chart, not all data that you might have in the database and which you did not load).
If the data was "cleaned" after market close, it's normal that the chart itself might change slightly and that will eventually generate different statistical results.
This is one of the problems of intraday data. Some feeds "clean" the data after market close to remove bad ticks. Exchanges themselves are known for cleaning their data.
If you trade stocks data is even worse because each exchange trades at different prices so one can have the daily high of AAPL in BATS at one price and in ARCA at another.

Since futures are centralized in one exchange this facilitates the data accuracy but then again, by "cleaning" the data at the day's end, it can be problematic.
You have several ways to minimize this. First, change datafeed. Go to IQFeed (if you only trade futures) since it's cheaper than eSignal. Also many users report that eSignal also "clean" their data, it's not 100% unfiltered.
IQFeed also might "clean" their data from time to time if a bad tick gets reported but apart from that, they maintain it complete unfiltered, as it came in. Used it for years and they are the best.
Another way is to not reload data in MC, keeping this way all intraday intact as it came in.

Thanks, arnie.
I am in contact with IQ-Feed. They provide a 24 hours guest account, so that I could test their data.

Regards
asyx