Chart mismatch Edit program done (moved to User contributed)

bowlesj3 · Postby **bowlesj3** » 26 Mar 2010

The scripts are done. moved to User contributed studies at this URL.
http://forum.tssupport.com/viewtopic.php?t=7286

======================================
In the thread you regularly see with subject

4th Subject: MC6 IB backfill spikes (history)

and with URL

http://forum.tssupport.com/viewtopic.php?t=6990

in the first few posts I thought that the charts were mismatching randomly and I made the comment below.

I guess I could write a matching records program to compare the charts (open high low close)

Regarding this concern, within the last week, I have noticed a random mismatch between the 1 minute bars and the 10 second bars and thus I have decided to write that chart mismatch edit program. I can do it now since I have my todo list cleared. I will make this edit program available for anyone to use.

Below are the initial ideas I have for specifications for this edit program. I will put this program and its documentation in the user contributed studies and make references to it in this thread as to its initial status so that thread does not get too messy. I will include it as an example of EL collections (list of lists) use in the Damage Boy thread on EL collections.

Code: Select all


BigChart to SmallChart Matching Specs:



You will specifiy a series of charts you want to have compared in pairs for accuracy relative to each other. If there is a mismatch you get a popup alert and report.



You will apply a fileappend study to every chart you want to edit. You are editing only one symbol. My charts have only one symbol (actually my one workspace has only one symbol) so this study is set up for one symbol only charts.



The study will detect the bar size and write out the bar information (open, high, low, close) to a file using the fileappend. Initially only high and low will be edited. Later I may enhance it to edit open and close. 



The bar size will be part of the file name (standardized format of some sort).



A chart matching study study will be applied to one chart. It will read in the files of high/low data and compare them in pairs as you specifiy. The ELcollections list-of-lists commands will be used extensively to read in these text files so this is a good study to review for learning these commands.



The matching study will also read a parameter file that tells it what pairs of charts to compare. It will do the comparison and produce a report of any mismatches between the charts. If there are problems it will issue an alert.



In intial example parameter file will be provided to give almost any combination of chart comares you can think of. You just remove the ones you do not need and maybe add one or two if needed.



If you get an alert you will go back to the charts to see if the report (which is written to C:\ root) is reporting the errors correctly.



Tick charts will not be included since I personally don't use them.



Daily and Weekly charts will not intially be included since I do not use the weekly charts. I only use daily charts to get a rough idea of what the bigger picture of the market is. Therefore daily to other chart compares will not be possible with this matching program (maybe later I will enhance it to do this)



Two options to run it:

1/ The study will be turned off during the day and turned on for offline edit after trading hours.

2/ You leave the studies on all day and at the end of the day (at the times you have entered in the two input parameters) the data for the day is written out for all charts and just after that the match program uses the parameters to run a match to compare all the charts you specifiy in pairs. You will get an alert if there were any mismatches across any charts.

Spaceant · Postby **Spaceant** » 26 Mar 2010

John,

Can't go to the URL link..... you may need to amend it.

Sa

Bruce DeVault · Postby **Bruce DeVault** » 26 Mar 2010

...I have noticed a random mismatch between the 1 minute bars and the 10 second bars...

I apologize in that I haven't been following all of these threads, but would just quickly add with the hope that it helps that most data feeds provide three native data types: daily, minute, and tick. Second bars are built from tick data, and there is no guarantee the tick data will match the minute data - in fact small discrepancies are more common than any of us would like. The reason for this often has to do with how timestamping is done or how bad ticks are filtered out - on some feeds for instance the minute data turns out to be cleaner, while the tick data more closely resembles the bad ticks etc. that are received in real-time. Only a very few data feeds supply "real-time minute data" and because of this, most trading platforms don't even use it when they do because they need to be data feed independent - most feeds supply real-time tick data e.g. an update every time there's a trade or inside bid/ask change, and historical daily, minute and tick data. Thus, almost all of the trading platforms build real-time minute data out of the real-time tick data, resulting in an even more common observation that there are sometimes differences between real-time minutes and historical minutes (since the real-time minutes are made from real-time tick data, and the tick data isn't guaranteed to match).

What you would expect to match 100% would be let's say X seconds vs. X * Y seconds, or other bars that are also built from tick data, instead of minute data which is a different native type all the way from the data feed.

Some platforms do offer the choice to make minute bars from tick data and disregard the data feed's native minute data type, and this typically increases consistency in matching things up, although at a cost of lower performance because the amount of data to be processed is hugely greater to achieve the same ends.

Does that make sense?

bowlesj3 · Postby **bowlesj3** » 26 Mar 2010

Spaceant: opps! I took the quote off the URL link to allow you to click on it.

Bruce: Interesting datafeed info. It makes sense I guess. I will still write the edit since I have some spare time on weekends now. Hopefully the only differences are between the second bars and the other charts. It will be worth the effort just to know there is for sure no problem.

Bruce DeVault · Postby **Bruce DeVault** » 26 Mar 2010

I think doing this type of comparison is an excellent idea, and look forward to seeing what you come up with. Just trying to shine a little more light on why those types of issues tend to happen. It could also be there's an issue in MC itself, but it may be that it's a function of the data feed issues described above, and the program you're writing will help clarify that so you can see more in detail what's different about it, and I think that's a good thing.

bowlesj3 · Postby **bowlesj3** » 26 Mar 2010

Yes for sure.

bowlesj3 · Postby **bowlesj3** » 26 Mar 2010

There will be one time at least that mismatches appear between the 1 minute bars and larger bar sizes such as 5 minute. That will be when I get those backfill spikes every few days. I will run it at those times before I fix them with the reloads as a way to test the edit program.

bowlesj3 · Postby **bowlesj3** » 26 Mar 2010

removed

bowlesj3 · Postby **bowlesj3** » 27 Mar 2010

removed

bowlesj3 · Postby **bowlesj3** » 28 Mar 2010

removed

bowlesj3 · Postby **bowlesj3** » 29 Mar 2010

Both options of the chart compare script are complete. I moved it to user contributed studies.

http://forum.tssupport.com/viewtopic.php?t=7286

bowlesj3 · Postby **bowlesj3** » 01 Apr 2010

Regarding what you were saying above Bruce (as quoted below)

I apologize in that I haven't been following all of these threads, but would just quickly add with the hope that it helps that most data feeds provide three native data types: daily, minute, and tick. Second bars are built from tick data, and there is no guarantee the tick data will match the minute data - in fact small discrepancies are more common than any of us would like. The reason for this often has to do with how timestamping is done or how bad ticks are filtered out - on some feeds for instance the minute data turns out to be cleaner, while the tick data more closely resembles the bad ticks etc. that are received in real-time. Only a very few data feeds supply "real-time minute data" and because of this, most trading platforms don't even use it when they do because they need to be data feed independent - most feeds supply real-time tick data e.g. an update every time there's a trade or inside bid/ask change, and historical daily, minute and tick data. Thus, almost all of the trading platforms build real-time minute data out of the real-time tick data, resulting in an even more common observation that there are sometimes differences between real-time minutes and historical minutes (since the real-time minutes are made from real-time tick data, and the tick data isn't guaranteed to match).

What you would expect to match 100% would be let's say X seconds vs. X * Y seconds, or other bars that are also built from tick data, instead of minute data which is a different native type all the way from the data feed.

Some platforms do offer the choice to make minute bars from tick data and disregard the data feed's native minute data type, and this typically increases consistency in matching things up, although at a cost of lower performance because the amount of data to be processed is hugely greater to achieve the same ends.

the new history report of results I am getting from the ChartEdit compare runs seem to match up with what you are saying. They can be found in the history report I just added. Immediately below is the current status of that history file. The numbers are for the data processed on the date to the left as of the run time shown right next to it. This is reporting mismatches between 1 minute bars and 10 second bars. 60 min, 30min, 15min, 10min, 5min and 1 minute bars match perfectly on all runs so far (unless there is a backfill spike error which I have not seen yet this week). I am not editing daily bars against any other bar types yet, although I suspect there may be problems there (but I am not concerned since I don't use daily bars that much).

Date 2010/Mar/31 Time 16:10 TotalMissmatchCnt 45 HighMissmatchCnt 25 LowMissmatchCnt 20
Date 2010/Apr/01 Time 08:55 TotalMissmatchCnt 38 HighMissmatchCnt 21 LowMissmatchCnt 17

Comments about the above two lines:
The first line is saying that on the 31st between midnight and 16:10 there were 45 of the 1-minute bars that did not have a perfect match in the 10-second bars. As you can see from the second line I did a run the next day at 8:55 and the results are very close. This suggests that most of these mismatches occur overnight when trading volume is fairly light (seems a bit backwards to me actually).

For anyone interested in doing these compares in their setup
The zip file in the thread
http://forum.tssupport.com/posting.php? ... st&p=32120
now contains
1/ The two scripts required for the edit (revised today)
2/ A sample comparison picture in png format (new today)
3/ The required input paramter for my setup.
4/ A sample detailed results report
5/ A sample history report of the final totals (new today - above)
The post explains it all.

I am going to try to edit daily bars this weekend.

Chart mismatch Edit program done (moved to User contributed)

Chart mismatch Edit program done (moved to User contributed)

Re: Chart mismatch Edit program to be written soon.