Discrepancy - IB data saved in QM Vs IB streaming T&S data

Questions about MultiCharts and user contributed studies.
hilbert
Posts: 224
Joined: 17 Aug 2011
Has thanked: 76 times
Been thanked: 64 times

Discrepancy - IB data saved in QM Vs IB streaming T&S data

Postby hilbert » 21 Nov 2014

I wanted to do a comparison of tick-trade data between IB time&sales window against the tick-trade data stored in QM in Multicharts, to confirm integrity of data in MC.

I opened MC, created a new workspace with a 2 minute chart of ESZ4. Selected the option 'Do not download missing data'. After streaming live data for '1 minute 16 seconds', I took a screenshot of TWS time and sales window and then closed MC, so that data gets stored in QM. I reopened MC and QM and exported trade-tick data from MC to an ASCII file to do a comparison against the screenshot of TWS time&sales window. Following are my observations:

1) There were 48 trade-ticks stored in MC Vs 40 trades displayed in IB's time&sales. Why are they not same?
2) On closer examination, I see that most of the trades with a +sign before them displayed in IB's time&sales were broken into multiple trades and stored in MC database. However interestingly, there were 2-3 cases, when multiple trades in IB time&sales were combined to give multiple (different trades) in MC, for the same number of contracts and at similar time. Overall, if we are getting same volume at correct prices, this is not a problem (because in any case, we are not getting true tick data from IB). However, following observations are a problem:

3) Out of a total of 772 contracts traded in this 1min 16sec inteval, there was a net discrepancy of 40 contracts in volume, which is a discrepancy of 5% between data displayed on IB time&sales and data stored in MC. Why should there be a discrepancy of even 1 contract? Remember, I had only 1 chart open with no indicators, so its not that there is any lag etc.
4) In 2 trades out of 48 trades, prices and trade times got interchanged between streaming data in IB time&sales and data stored in MC. This should not happen. This means there is a problem in the way MC stores data. I encourage MC team to tell us more about why such data interchange happens, when in most other cases, its fine?
5) There was a clearly erroneous (false tick) of 33 contracts stored in MC. It should not have been there (green colored). Why this ghost tick came in the MC? This is a big problem.

I fully understand that IB displays snapshot data at 250ms or 300ms interval. However, I don't understand why data stored in MC should be any different from IB's streaming time&sales data. Is there a known/unknown bug or a limitation in the way MC processes and stores data from IB? Are other charting programs better in this regard compared to MC? I welcome all comments (please back your assertion with real test results) from other users, if you have access to other programs.

I am also concerned about how the data quality will be impacted when storing data from 50 charts simultaneously instead of using just 1 chart.
Attachments
Ticks stored in MC.txt
(2.27 KiB) Downloaded 441 times
IB TWS T&S Vs MC QM tick data.png
(160.64 KiB) Downloaded 414 times

User avatar
Henry MultiСharts
Posts: 9165
Joined: 25 Aug 2011
Has thanked: 1264 times
Been thanked: 2957 times

Re: Discrepancy - IB data saved in QM Vs IB streaming T&S da

Postby Henry MultiСharts » 21 Nov 2014

Hello hilbert,

That is not relevant to compare Time and Sales data from TWS with MultiCharts chart data. These are two different sources of data. Even Time and Sales data in TWS can be different from Time and Sales data in MultiCharts not only because MultiCharts calculates the Time and Sales internally and our formula can be different from TWS, but also because there have been enhancements made for IB TWS Time & Sales which now provides additional settings and ticks exposed to TWS that are not mapped to the API.

If you want to get pure data streamed by IB API - please disable the option “Generate a new tick if Total Volume changes” in QuoteManager -> Tools Data Sources -> Interactive Brokers. Then restart MultiCharts and all of its processes

When it's done - MultiCharts data can be compared to the "IB Example" application data (which shows exactly what is sent via the API connection, and which can be different from TWS). If you want to run the "IB Example" application - please contact us directly.

hilbert
Posts: 224
Joined: 17 Aug 2011
Has thanked: 76 times
Been thanked: 64 times

Re: Discrepancy - IB data saved in QM Vs IB streaming T&S da

Postby hilbert » 21 Nov 2014

Hello hilbert,

That is not relevant to compare Time and Sales data from TWS with MultiCharts chart data. These are two different sources of data. Even Time and Sales data in TWS can be different from Time and Sales data in MultiCharts not only because MultiCharts calculates the Time and Sales internally and our formula can be different from TWS, but also because there have been enhancements made for IB TWS Time & Sales which now provides additional settings and ticks exposed to TWS that are not mapped to the API.
Hey Henry, Thanks for the information. 2 quick questions:
1) What do you mean by "MC calculates Time&Sales internally and our formula can be different from TWS"? This is just streaming data, not a formula/indicator/strategy that can be calculated. I thought MC just catches the streaming data from IB API, and displays it in MC. I apologize if I am missing something obvious here.

2) When did IB enhance TWS Time&Sales to provide additional settings and ticks exposed to TWS that are not mapped to API? Can you please point me to this announcement. I will be thankful.

I will disable "Generate a new tick if total volume changes" setting and test again. Btw, I found another thread from 2011 which talks about the same issue, but it remained unresolved.
viewtopic.php?f=1&t=8314

User avatar
Henry MultiСharts
Posts: 9165
Joined: 25 Aug 2011
Has thanked: 1264 times
Been thanked: 2957 times

Re: Discrepancy - IB data saved in QM Vs IB streaming T&S da

Postby Henry MultiСharts » 21 Nov 2014

Hey Henry, Thanks for the information. 2 quick questions:
1) What do you mean by "MC calculates Time&Sales internally and our formula can be different from TWS"? This is just streaming data, not a formula/indicator/strategy that can be calculated. I thought MC just catches the streaming data from IB API, and displays it in MC. I apologize if I am missing something obvious here.
Time&Sales is being calculated by MultiCharts based on the data supplied by the IB API. It does not come differently from IB API as a ready Time&Sales data that can be plotted rightaway.
2) When did IB enhance TWS Time&Sales to provide additional settings and ticks exposed to TWS that are not mapped to API? Can you please point me to this announcement. I will be thankful.
We were informed that it has been done during the last couple of years. We have received this information from IB tech support directly.
I will disable "Generate a new tick if total volume changes" setting and test again. Btw, I found another thread from 2011 which talks about the same issue, but it remained unresolved. viewtopic.php?f=1&t=8314
This is not something on our end - the same data should be provided to the TWS and API by IB.


Return to “MultiCharts”