Hello all
I have posted about this before but there was always a little tweak to make the difference smaller.
I have now exhausted the established fixes for this problem and I wanted to post this here to share my thoughts and seek advice.
Below are screen shots of exported data from MC.
IQFeed is my data source. I had IB before and there were even more problems.
My IQFeed settings are all unchecked. No filtering of any kind.
Here are the exact steps I take to recreate these problems.
Open MC
Open 2 charts exactly the same. I chose Regular Trading Hours. 5 min Bars. ticker = SPY
Let them both run all day long.
At the end of the day I change the time frame of the second chart to 10 min and then back to 5 min.
Now I go to File>Export Data
I export both charts and run a diff check on them.
These pictures are those diff checks.
The Volume is off on several bars.
~~~~~~~
I have been in contact with both MC and IQFeed. Neither of them knew about this problem?
They have both been very helpful but have not solved this problem.
I believe that neither of them knew the problem existed before I started my inquiry.
How could you possibly train and optimize on a data set that is <> to the original data set that first happened there?
If you use volume in your equation then you will absolutely have trades that are false positives and false negatives at some point. I know I do on a very regular basis. Especially in High volatility.
I can email you all the relative data but its not hard to see it yourself.
Put a VWAP on a 5 min chart. Let the chart run all day then refresh it. The VWAP will have different values.
Historical vs Streaming Analysis
Historical vs Streaming Analysis
- Attachments
-
- Diff check 12-21-18.PNG
- (209.59 KiB) Downloaded 532 times
-
- image.png
- (162.43 KiB) Downloaded 532 times
- TJ
- Posts: 7743
- Joined: 29 Aug 2006
- Location: Global Citizen
- Has thanked: 1033 times
- Been thanked: 2223 times
Re: Historical vs Streaming Analysis
Have you compared your data with the exchange's data?
I haven't and that seems like a good place to start.
But..
if the exchange matches historical but not streaming then I still don't have a solution.
- TJ
- Posts: 7743
- Joined: 29 Aug 2006
- Location: Global Citizen
- Has thanked: 1033 times
- Been thanked: 2223 times
Re: Historical vs Streaming Analysis
Historic data is the bench.I haven't and that seems like a good place to start.Have you compared your data with the exchange's data?
But..
if the exchange matches historical but not streaming then I still don't have a solution.
Streaming data is always different. ALWAYS. It is the same with all the data providers.
Re: Historical vs Streaming Analysis
Historic data is the bench.I haven't and that seems like a good place to start.Have you compared your data with the exchange's data?
But..
if the exchange matches historical but not streaming then I still don't have a solution.
Streaming data is always different. ALWAYS. It is the same with all the data providers.
And that would be fine.
If the sales and support for services admitted this. IQFeed is absolutely positive that the streaming data should always be the same as historical. MC support says they don't know any reason streaming is different than Historical. TS support says there is no way streaming will be different than
historical. Even Nanex says their data is set in stone after the time has passed.
So where is the disconnect. All these providers promising a stream that is equal to the historical and none provide it?
what is the limit to this? how often can updates be made? when are they made? Is there a data provider selling live stream data only..... cause lets be honest thats the only data set that matters?
one edit...
IB support says streaming is completely different than historical
- TJ
- Posts: 7743
- Joined: 29 Aug 2006
- Location: Global Citizen
- Has thanked: 1033 times
- Been thanked: 2223 times
Re: Historical vs Streaming Analysis
Historic data is the bench.I haven't and that seems like a good place to start.
But..
if the exchange matches historical but not streaming then I still don't have a solution.
Streaming data is always different. ALWAYS. It is the same with all the data providers.
And that would be fine.
If the sales and support for services admitted this. IQFeed is absolutely positive that the streaming data should always be the same as historical. MC support says they don't know any reason streaming is different than Historical. TS support says there is no way streaming will be different than
historical. Even Nanex says their data is set in stone after the time has passed.
So where is the disconnect. All these providers promising a stream that is equal to the historical and none provide it?
what is the limit to this? how often can updates be made? when are they made? Is there a data provider selling live stream data only..... cause lets be honest thats the only data set that matters?
one edit...
IB support says streaming is completely different than historical
The disconnect is at your computer.
Data is data . . . there is no 5 min charts or hourly charts.
Data is just a continuous stream.
The 5 min chart is YOUR creation at your computer.
The time delineation is created by you.
The micro-second when the tick appears on your screen, it is ancient history.
Many new ticks have already happened at the exchange far away from you, at the same time.
Re: Historical vs Streaming Analysis
Historic data is the bench.
Streaming data is always different. ALWAYS. It is the same with all the data providers.
And that would be fine.
If the sales and support for services admitted this. IQFeed is absolutely positive that the streaming data should always be the same as historical. MC support says they don't know any reason streaming is different than Historical. TS support says there is no way streaming will be different than
historical. Even Nanex says their data is set in stone after the time has passed.
So where is the disconnect. All these providers promising a stream that is equal to the historical and none provide it?
what is the limit to this? how often can updates be made? when are they made? Is there a data provider selling live stream data only..... cause lets be honest thats the only data set that matters?
one edit...
IB support says streaming is completely different than historical
The disconnect is at your computer.
Data is data . . . there is no 5 min charts or hourly charts.
Data is just a continuous stream.
The 5 min chart is YOUR creation at your computer.
The time delineation is created by you.
The micro-second when the tick appears on your screen, it is ancient history.
Many new ticks have already happened at the exchange far away from you, at the same time.
So your opinion is that if I were a computer sitting right next to DTN IQFeed then I wouldn't have this problem of streaming vs Historical?
Do you think this is a problem at all? I honestly feel like you are offended that I would even suggest I receive data matching the historical data?
If this is a problem on my end how do I fix it? Am I really the first person to ask the question... why am I in a trade that my chart says didn't happen yesterday?
- TJ
- Posts: 7743
- Joined: 29 Aug 2006
- Location: Global Citizen
- Has thanked: 1033 times
- Been thanked: 2223 times
Re: Historical vs Streaming Analysis
No, I am not offended by your question. This has been asked many times.
Think about the distance the tick has to travel from the exchange to your computer:
Exchange-->internet provider-->IQFeed-->internet provider-->your router-->your computer
You can do a PING test to check your latency.
ps. Also look up Aggregated tick, Snapshot.
(IB website has very good explanations.)
Think about the distance the tick has to travel from the exchange to your computer:
Exchange-->internet provider-->IQFeed-->internet provider-->your router-->your computer
You can do a PING test to check your latency.
ps. Also look up Aggregated tick, Snapshot.
(IB website has very good explanations.)
Re: Historical vs Streaming Analysis
I have looked through the posts here associated with Aggregated tick. Some of them were mine.
They were helpful.
And your help along with Zheka was appreciated.
Questions
Does this problem affect your trading on a regular basis? Is it just my particular strategy that experiences erroneous results?
If I have 2 computers in two separate places would you expect there errors to be different?
If this is just a send receive error. Wouldn't an obvious fix be to double check the on chart results on a regular systematic basis to ensure correctness?
They were helpful.
And your help along with Zheka was appreciated.
Questions
Does this problem affect your trading on a regular basis? Is it just my particular strategy that experiences erroneous results?
If I have 2 computers in two separate places would you expect there errors to be different?
If this is just a send receive error. Wouldn't an obvious fix be to double check the on chart results on a regular systematic basis to ensure correctness?
- TJ
- Posts: 7743
- Joined: 29 Aug 2006
- Location: Global Citizen
- Has thanked: 1033 times
- Been thanked: 2223 times
Re: Historical vs Streaming Analysis
If you use a magnifying glass, you see details.
If you use a microscope, you see more details.
If you use an electron microscope, you see even more details.
If you found yourself seeing too much detail,
you should take a step back, and look again.
Different computers can show different results,
not because things are different,
but because things were never promised to be the same.
If you use a microscope, you see more details.
If you use an electron microscope, you see even more details.
If you found yourself seeing too much detail,
you should take a step back, and look again.
Different computers can show different results,
not because things are different,
but because things were never promised to be the same.
Re: Historical vs Streaming Analysis
"The Devil is in the details"
the details of a matter are its most problematic aspect.
Yeah, I agree that I could redesign the strategy for a potentially more robust view of Volume.
Your answer is elegant but the reason I chose to program my trades is to measure things... Precisely. As precise as I possibly can.
So by "unmeasuring" things it feels like a step back not a step forward.
And your last line of "things were never promised to be the same" is not really the case. They were promised by support and customer service to be the same EVERY time I contacted them. So it appears that you and I are the only people who know they aren't.
And to be fair to MC this has nothing to do with them. Its a data provider thing. Although you would think that someone at MC would know that this is happening.
the details of a matter are its most problematic aspect.
Yeah, I agree that I could redesign the strategy for a potentially more robust view of Volume.
Your answer is elegant but the reason I chose to program my trades is to measure things... Precisely. As precise as I possibly can.
So by "unmeasuring" things it feels like a step back not a step forward.
And your last line of "things were never promised to be the same" is not really the case. They were promised by support and customer service to be the same EVERY time I contacted them. So it appears that you and I are the only people who know they aren't.
And to be fair to MC this has nothing to do with them. Its a data provider thing. Although you would think that someone at MC would know that this is happening.
Re: Historical vs Streaming Analysis
Hi Joebone,
Bars built from ticks arriving at your computer can’t be expected to exactly match historical bars from the data provider due to the nature of the internet, routing, computer system time, etc, and the differences you’ve observed become more apparent as the resolution gets higher. I think this is why wholesale traders pay premiums to get their servers right on the trading floor? I’m not sure there’s anything you can do about it.
Bars built from ticks arriving at your computer can’t be expected to exactly match historical bars from the data provider due to the nature of the internet, routing, computer system time, etc, and the differences you’ve observed become more apparent as the resolution gets higher. I think this is why wholesale traders pay premiums to get their servers right on the trading floor? I’m not sure there’s anything you can do about it.
Re: Historical vs Streaming Analysis
IQFeed has confirmed that they see the problem exists and are analyzing solutions.
From Anna at MC
So we received the reply from Steven from IQFeed Developer Support, here it is:
“I believe I have found the source of this issue although I haven't been able to confirm it yet. I have been able to confirm that the streaming data is wrong and the historical data is correct in this case."
~~~~
Cheers
From Anna at MC
So we received the reply from Steven from IQFeed Developer Support, here it is:
“I believe I have found the source of this issue although I haven't been able to confirm it yet. I have been able to confirm that the streaming data is wrong and the historical data is correct in this case."
~~~~
Cheers
Re: Historical vs Streaming Analysis
Well Svetlana has provided a possible fix for this problem for now.
I haven't tested this yet... so its unconfirmed
command.rld
https://www.multicharts.com/trading-sof ... ommandLine
There is no way to backtest this fix without a record of the streamed data. But at least I can be confident that my chart is as up to date as possible....
Hope this helps everyone struggling with this problem. and thanks Svetlana
I haven't tested this yet... so its unconfirmed
command.rld
https://www.multicharts.com/trading-sof ... ommandLine
There is no way to backtest this fix without a record of the streamed data. But at least I can be confident that my chart is as up to date as possible....
Hope this helps everyone struggling with this problem. and thanks Svetlana