Streaming vs Historical Data  [SOLVED]

Hello all,

First post here at MC. I came from ToS and really enjoyed backtesting and coding in thinkscript. Upgraded to MC and IB so I can finally put my ideas into action with automated trading.

Something has come to my attention that i find very troubling.

I have a signal that I use on 5 min bars in SPY. IB broker and IB data source.

I run it only on regular trading hours in AA mode for now. So I shut down the software and the Gateway at the end of the day.

I have found that after loading up everything before the market opens and turning on Signals, occasionally there is a signal to open or close that wasn't there yesterday when I shut down the software. The first time I thought it was a rare occurrence. It happened again within 2 weeks. I talked to support and they said that sometimes data providers "scrub" or "fix" data for historical bars....

this is extremely alarming.... how am I supposed to write code to trade streaming data if streaming_data <> historical_data?

obviously many of you have found this not to be a complete disaster and manage to work around it?

what are common practices? Do you have to code with this in mind? would you slot this under a robustness check? what are the best data sources for this not to have an effect?

thanks

Great question.

This is a very important concept. For me historical==streaming otherwise all my research is being undermined!

I have no issues using IQfeed re historical vs streaming fidelity.

I did have issues with CQG realtime data and so stopped using it.

When I periodically compile my datasets for reoptimisation cycles I will find small differences in open and close prints when comparing TS, IQfeed and CQGs data. Point being; a small degree of uncertainty is unavoidable it seems.

But....your issue would drive me nuts. Think about using IQfeed maybe? Whatever solution you find, please please please share it here!

Thanks.

IB does not stream tick-by-tick data.

They use aggregated ticks. You can read more about it on their website.

ps. most data provider also uses aggregated ticks to various degree. Please read the fine prints.

Great question.

This is a very important concept. For me historical==streaming otherwise all my research is being undermined!

I have no issues using IQfeed re historical vs streaming fidelity.

I did have issues with CQG realtime data and so stopped using it.

When I periodically compile my datasets for reoptimisation cycles I will find small differences in open and close prints when comparing TS, IQfeed and CQGs data. Point being; a small degree of uncertainty is unavoidable it seems.

But....your issue would drive me nuts. Think about using IQfeed maybe? Whatever solution you find, please please please share it here!

Thanks.

I'm glad I'm not the only one that is blown away by this.. and I'm honestly surprised it isn't a created thread already....

I don't have a solution yet, unfortunately. I thought surely this is a real issue and everyone here has had to deal with it so there should be a laundry list of solutions.... I haven't found any yet.

Update...

I had another trade going into the close last night that wasn't there when i shut down the software that is there this morning.... Maybe my system is particularly susceptible to this?? I don't know yet. Hopefully someone can shed some light on this issue.

IB does not stream tick-by-tick data.

They use aggregated ticks. You can read more about it on their website.

ps. most data provider also uses aggregated ticks to various degree. Please read the fine prints.

yes, I talked to MC support about this and they explain that part. That doesn't address the streaming data <> historical data... does it?

I'm satisfied with my IQfeed solution.

But, 100% agree with you on your points.

For me the most important parts of systematic trading are:

1st: integrity and accuracy of datasets (both for research and live trading purposes).

2nd: accuracy of historic simulations.

3rd: fidelity of live trading (it needing to be as close to 100% of how the historical simulator sees it)

Then everything else.....

So I, like you, find it baffling that there is so much focus on the bells and whistles and not on what's important (I'm in a ranting mood today).

This feels like something i could rant about... how is it not at the top of everyone list..

maybe its because I'm using IB for data... this really has nothing to do with MC and everything to do with data provider

So maybe the simple question is what is everyone data provider?

Great question.
This is a very important concept. For me historical==streaming otherwise all my research is being undermined!
I have no issues using IQfeed re historical vs streaming fidelity.
I did have issues with CQG realtime data and so stopped using it.
When I periodically compile my datasets for reoptimisation cycles I will find small differences in open and close prints when comparing TS, IQfeed and CQGs data. Point being; a small degree of uncertainty is unavoidable it seems.
But....your issue would drive me nuts. Think about using IQfeed maybe? Whatever solution you find, please please please share it here!
Thanks.

I'm glad I'm not the only one that is blown away by this.. and I'm honestly surprised it isn't a created thread already....

. . .

There are multiple threads on this subject already, with detailed explanations.

IB does not stream tick-by-tick data.

They use aggregated ticks. You can read more about it on their website.

ps. most data provider also uses aggregated ticks to various degree. Please read the fine prints.

yes, I talked to MC support about this and they explain that part. That doesn't address the streaming data <> historical data... does it?

Please see IB's website.

(or the many threads in this forum.)

IB does not stream tick-by-tick data.

They use aggregated ticks. You can read more about it on their website.

ps. most data provider also uses aggregated ticks to various degree. Please read the fine prints.

yes, I talked to MC support about this and they explain that part. That doesn't address the streaming data <> historical data... does it?

Please see IB's website.

(or the many threads in this forum.)

Thanks TJ

I don't know how i missed them. Ill look again

some discussion here
viewtopic.php?f=19&t=46221&p=103276&hil ... ta#p103276

THE solution to this issue is to use IB's True Real-Time 5-sec bars, instead of ticks.
For that, switch to "use 5-sec bars as ticks" in IB's plug-in(datasource) settings; you might need to use Trades as a datastream (rather than Bid-Ask).
Please search forum for related/my posts for details.

Search also these keywords:

aggregated ticks
snapshot
combined quotes
IB
stream
dataprovider

etc.,

thanks very much for the directions for further research

THE solution to this issue is to use IB's True Real-Time 5-sec bars, instead of ticks.
For that, switch to "use 5-sec bars as ticks" in IB's plug-in(datasource) settings; you might need to use Trades as a datastream (rather than Bid-Ask).
Please search forum for related/my posts for details.

I attached a screen shot of my Data Feed Configuration

is the the "Use 5 seconds bars as ticks" you are talking about

again thanks for help

is the the "Use 5 seconds bars as ticks" you are talking about

Yes, this is the one. 5-sec bars come with 250-300ms delay in real-time from IB's historical data server, cleaned, so you are guaranteed to have historical=real-time. All with minor processing load compared to getting all of the ticks in RT.
Set data to Trades and 1-tick and see how frequently MC plots/updates the data to ensure it actually works.
It use it with FX, did not try with futures.

is the the "Use 5 seconds bars as ticks" you are talking about

Yes, this is the one. 5-sec bars come with 250-300ms delay in real-time from IB's historical data server, cleaned, so you are guaranteed to have historical=real-time. All with minor processing load compared to getting all of the ticks in RT.
Set data to Trades and 1-tick and see how frequently MC plots/updates the data to ensure it actually works.
It use it with FX, did not try with futures.

Is this the "set data to trades" ?

do i need to use tick resolutions or can i use minute bars?

@ Zheka

also should this solution work for stocks as well?

Set data to Trades and 1-tick and see how frequently MC plots/updates the data to ensure it actually works.

My suggestion re 1-tick resolution was to VERIFY (rather than ensure) that it works.

These 5-sec bars will be the "ticks" (actually, they are recorded by MC as 4-ticks in the database - O-H-L-C) from which all other resolutions will be built in real-time.

It should work for stocks. Just try...

Set data to Trades and 1-tick and see how frequently MC plots/updates the data to ensure it actually works.

My suggestion re 1-tick resolution was to VERIFY (rather than ensure) that it works.

These 5-sec bars will be the "ticks" (actually, they are recorded by MC as 4-ticks in the database - O-H-L-C) from which all other resolutions will be built in real-time.

It should work for stocks. Just try...

Excellent, thanks very much! There's a lot learn in this arena and trading is just 1 tiny part.... your help is appreciated.

Zheka, TJ

I am using the 5 sec tick and had another false negative yesterday into the close. I am now looking at an exit trade 15;45 and there wasn't one there yesterday. It seems this is the time that it most likely fails. That must be the time when significant adjustments are made on the IB servers cause of increased volume and market participants.

I know you have both helped me already so I wasn't complaining. Just wanted to leave a note here in the forums that the problem persists

thanks

Well, that's as close as you can get it...And it is indeed possible that some trades were busted/cleaned out and corrections made after the day close.
No one is to blame, that's just "real-life".

The best you can do is to "Flush cached data to database" before closing MC.
So, MC should/will not be re-loading historical data for the past.
There might be differences in your system trades vs. "pure historical data", but they should even out over time...

Yeah, I figured.. thanks again for reply.

Since it only seems to happen on around the close I'm thinking i will just load up a fresh chart around 4:15-4:20 every day and make sure I didnt get a false negative. Hopefully it wont skew my results to much... fortunately this AM I was long overnight so that was a blessing....

Again.. thanks for all your help

Here is a possible solution to this problem

I haven't tested this yet... so its unconfirmed
command.rld
https://www.multicharts.com/trading-sof ... ommandLine

This will not work reliably all the time. I could not make 'data reloading' work well in live trading..

This will not work reliably all the time. I could not make 'data reloading' work well in live trading..

just when i had hope... what was the problem?