Streaming vs Historical Data  [SOLVED]

Questions about MultiCharts and user contributed studies.
User avatar
joebone
Posts: 175
Joined: Sep 05 2018
Has thanked: 53 times
Been thanked: 4 times

Sep 12 2018

Hello all,

First post here at MC. I came from ToS and really enjoyed backtesting and coding in thinkscript. Upgraded to MC and IB so I can finally put my ideas into action with automated trading.

Something has come to my attention that i find very troubling.

I have a signal that I use on 5 min bars in SPY. IB broker and IB data source.

I run it only on regular trading hours in AA mode for now. So I shut down the software and the Gateway at the end of the day.

I have found that after loading up everything before the market opens and turning on Signals, occasionally there is a signal to open or close that wasn't there yesterday when I shut down the software. The first time I thought it was a rare occurrence. It happened again within 2 weeks. I talked to support and they said that sometimes data providers "scrub" or "fix" data for historical bars....

this is extremely alarming.... how am I supposed to write code to trade streaming data if streaming_data <> historical_data?

obviously many of you have found this not to be a complete disaster and manage to work around it?

what are common practices? Do you have to code with this in mind? would you slot this under a robustness check? what are the best data sources for this not to have an effect?

thanks

wilkinsw
Posts: 662
Joined: Apr 21 2013
Has thanked: 154 times
Been thanked: 104 times

Sep 13 2018

Great question.

This is a very important concept. For me historical==streaming otherwise all my research is being undermined!

I have no issues using IQfeed re historical vs streaming fidelity.

I did have issues with CQG realtime data and so stopped using it.

When I periodically compile my datasets for reoptimisation cycles I will find small differences in open and close prints when comparing TS, IQfeed and CQGs data. Point being; a small degree of uncertainty is unavoidable it seems.

But....your issue would drive me nuts. Think about using IQfeed maybe? Whatever solution you find, please please please share it here!

Thanks.

User avatar
TJ
Posts: 7763
Joined: Aug 29 2006
Location: Global Citizen
Has thanked: 1036 times
Been thanked: 2232 times

Sep 13 2018

IB does not stream tick-by-tick data.

They use aggregated ticks. You can read more about it on their website.

ps. most data provider also uses aggregated ticks to various degree. Please read the fine prints.

User avatar
joebone
Posts: 175
Joined: Sep 05 2018
Has thanked: 53 times
Been thanked: 4 times

Sep 13 2018

Great question.

This is a very important concept. For me historical==streaming otherwise all my research is being undermined!

I have no issues using IQfeed re historical vs streaming fidelity.

I did have issues with CQG realtime data and so stopped using it.

When I periodically compile my datasets for reoptimisation cycles I will find small differences in open and close prints when comparing TS, IQfeed and CQGs data. Point being; a small degree of uncertainty is unavoidable it seems.

But....your issue would drive me nuts. Think about using IQfeed maybe? Whatever solution you find, please please please share it here!

Thanks.

I'm glad I'm not the only one that is blown away by this.. and I'm honestly surprised it isn't a created thread already....

I don't have a solution yet, unfortunately. I thought surely this is a real issue and everyone here has had to deal with it so there should be a laundry list of solutions.... I haven't found any yet.

User avatar
joebone
Posts: 175
Joined: Sep 05 2018
Has thanked: 53 times
Been thanked: 4 times

Sep 13 2018

Update...

I had another trade going into the close last night that wasn't there when i shut down the software that is there this morning.... Maybe my system is particularly susceptible to this?? I don't know yet. Hopefully someone can shed some light on this issue.

User avatar
joebone
Posts: 175
Joined: Sep 05 2018
Has thanked: 53 times
Been thanked: 4 times

Sep 13 2018

IB does not stream tick-by-tick data.

They use aggregated ticks. You can read more about it on their website.

ps. most data provider also uses aggregated ticks to various degree. Please read the fine prints.

yes, I talked to MC support about this and they explain that part. That doesn't address the streaming data <> historical data... does it?

wilkinsw
Posts: 662
Joined: Apr 21 2013
Has thanked: 154 times
Been thanked: 104 times

Sep 13 2018

I'm satisfied with my IQfeed solution.

But, 100% agree with you on your points.

For me the most important parts of systematic trading are:

1st: integrity and accuracy of datasets (both for research and live trading purposes).

2nd: accuracy of historic simulations.

3rd: fidelity of live trading (it needing to be as close to 100% of how the historical simulator sees it)

Then everything else.....

So I, like you, find it baffling that there is so much focus on the bells and whistles and not on what's important (I'm in a ranting mood today).

User avatar
joebone
Posts: 175
Joined: Sep 05 2018
Has thanked: 53 times
Been thanked: 4 times

Sep 13 2018

This feels like something i could rant about... how is it not at the top of everyone list..

maybe its because I'm using IB for data... this really has nothing to do with MC and everything to do with data provider

So maybe the simple question is what is everyone data provider?

User avatar
TJ
Posts: 7763
Joined: Aug 29 2006
Location: Global Citizen
Has thanked: 1036 times
Been thanked: 2232 times

Sep 13 2018

Great question.
This is a very important concept. For me historical==streaming otherwise all my research is being undermined!
I have no issues using IQfeed re historical vs streaming fidelity.
I did have issues with CQG realtime data and so stopped using it.
When I periodically compile my datasets for reoptimisation cycles I will find small differences in open and close prints when comparing TS, IQfeed and CQGs data. Point being; a small degree of uncertainty is unavoidable it seems.
But....your issue would drive me nuts. Think about using IQfeed maybe? Whatever solution you find, please please please share it here!
Thanks.

I'm glad I'm not the only one that is blown away by this.. and I'm honestly surprised it isn't a created thread already....

. . .

There are multiple threads on this subject already, with detailed explanations.

User avatar
TJ
Posts: 7763
Joined: Aug 29 2006
Location: Global Citizen
Has thanked: 1036 times
Been thanked: 2232 times

Sep 13 2018

IB does not stream tick-by-tick data.

They use aggregated ticks. You can read more about it on their website.

ps. most data provider also uses aggregated ticks to various degree. Please read the fine prints.

yes, I talked to MC support about this and they explain that part. That doesn't address the streaming data <> historical data... does it?

Please see IB's website.

(or the many threads in this forum.)

User avatar
joebone
Posts: 175
Joined: Sep 05 2018
Has thanked: 53 times
Been thanked: 4 times

Sep 13 2018

IB does not stream tick-by-tick data.

They use aggregated ticks. You can read more about it on their website.

ps. most data provider also uses aggregated ticks to various degree. Please read the fine prints.

yes, I talked to MC support about this and they explain that part. That doesn't address the streaming data <> historical data... does it?

Please see IB's website.

(or the many threads in this forum.)
Thanks TJ

I don't know how i missed them. Ill look again

User avatar
joebone
Posts: 175
Joined: Sep 05 2018
Has thanked: 53 times
Been thanked: 4 times

Sep 13 2018


Zheka
Posts: 225
Joined: Jan 13 2016
Has thanked: 8 times
Been thanked: 53 times

Sep 13 2018

THE solution to this issue is to use IB's True Real-Time 5-sec bars, instead of ticks.
For that, switch to "use 5-sec bars as ticks" in IB's plug-in(datasource) settings; you might need to use Trades as a datastream (rather than Bid-Ask).
Please search forum for related/my posts for details.

User avatar
TJ
Posts: 7763
Joined: Aug 29 2006
Location: Global Citizen
Has thanked: 1036 times
Been thanked: 2232 times

Sep 13 2018

Search also these keywords:

aggregated ticks
snapshot
combined quotes
IB
stream
dataprovider

etc.,

User avatar
joebone
Posts: 175
Joined: Sep 05 2018
Has thanked: 53 times
Been thanked: 4 times

Sep 13 2018

thanks very much for the directions for further research

User avatar
joebone
Posts: 175
Joined: Sep 05 2018
Has thanked: 53 times
Been thanked: 4 times

Sep 13 2018

THE solution to this issue is to use IB's True Real-Time 5-sec bars, instead of ticks.
For that, switch to "use 5-sec bars as ticks" in IB's plug-in(datasource) settings; you might need to use Trades as a datastream (rather than Bid-Ask).
Please search forum for related/my posts for details.
I attached a screen shot of my Data Feed Configuration

is the the "Use 5 seconds bars as ticks" you are talking about

again thanks for help
Attachments
5 sec ticks.PNG
(38.05 KiB) Downloaded 1743 times

Zheka
Posts: 225
Joined: Jan 13 2016
Has thanked: 8 times
Been thanked: 53 times

Sep 14 2018

is the the "Use 5 seconds bars as ticks" you are talking about
Yes, this is the one. 5-sec bars come with 250-300ms delay in real-time from IB's historical data server, cleaned, so you are guaranteed to have historical=real-time. All with minor processing load compared to getting all of the ticks in RT.
Set data to Trades and 1-tick and see how frequently MC plots/updates the data to ensure it actually works.
It use it with FX, did not try with futures.

User avatar
joebone
Posts: 175
Joined: Sep 05 2018
Has thanked: 53 times
Been thanked: 4 times

Sep 14 2018

is the the "Use 5 seconds bars as ticks" you are talking about
Yes, this is the one. 5-sec bars come with 250-300ms delay in real-time from IB's historical data server, cleaned, so you are guaranteed to have historical=real-time. All with minor processing load compared to getting all of the ticks in RT.
Set data to Trades and 1-tick and see how frequently MC plots/updates the data to ensure it actually works.
It use it with FX, did not try with futures.

Is this the "set data to trades" ?

do i need to use tick resolutions or can i use minute bars?
Attachments
data = trade.PNG
(20.87 KiB) Downloaded 1719 times

User avatar
joebone
Posts: 175
Joined: Sep 05 2018
Has thanked: 53 times
Been thanked: 4 times

Sep 14 2018

@ Zheka

also should this solution work for stocks as well?

Zheka
Posts: 225
Joined: Jan 13 2016
Has thanked: 8 times
Been thanked: 53 times

Sep 14 2018

Set data to Trades and 1-tick and see how frequently MC plots/updates the data to ensure it actually works.
My suggestion re 1-tick resolution was to VERIFY (rather than ensure) that it works.

These 5-sec bars will be the "ticks" (actually, they are recorded by MC as 4-ticks in the database - O-H-L-C) from which all other resolutions will be built in real-time.

It should work for stocks. Just try...

User avatar
joebone
Posts: 175
Joined: Sep 05 2018
Has thanked: 53 times
Been thanked: 4 times

Sep 14 2018

Set data to Trades and 1-tick and see how frequently MC plots/updates the data to ensure it actually works.
My suggestion re 1-tick resolution was to VERIFY (rather than ensure) that it works.

These 5-sec bars will be the "ticks" (actually, they are recorded by MC as 4-ticks in the database - O-H-L-C) from which all other resolutions will be built in real-time.

It should work for stocks. Just try...

Excellent, thanks very much! There's a lot learn in this arena and trading is just 1 tiny part.... your help is appreciated.

User avatar
joebone
Posts: 175
Joined: Sep 05 2018
Has thanked: 53 times
Been thanked: 4 times

Oct 03 2018

Zheka, TJ

I am using the 5 sec tick and had another false negative yesterday into the close. I am now looking at an exit trade 15;45 and there wasn't one there yesterday. It seems this is the time that it most likely fails. That must be the time when significant adjustments are made on the IB servers cause of increased volume and market participants.

I know you have both helped me already so I wasn't complaining. Just wanted to leave a note here in the forums that the problem persists

thanks

Zheka
Posts: 225
Joined: Jan 13 2016
Has thanked: 8 times
Been thanked: 53 times

Oct 03 2018

Well, that's as close as you can get it...And it is indeed possible that some trades were busted/cleaned out and corrections made after the day close.
No one is to blame, that's just "real-life".

The best you can do is to "Flush cached data to database" before closing MC.
So, MC should/will not be re-loading historical data for the past.
There might be differences in your system trades vs. "pure historical data", but they should even out over time...
Last edited by Zheka on Oct 03 2018, edited 1 time in total.

User avatar
joebone
Posts: 175
Joined: Sep 05 2018
Has thanked: 53 times
Been thanked: 4 times

Oct 03 2018

Yeah, I figured.. thanks again for reply.

Since it only seems to happen on around the close I'm thinking i will just load up a fresh chart around 4:15-4:20 every day and make sure I didnt get a false negative. Hopefully it wont skew my results to much... fortunately this AM I was long overnight so that was a blessing....

Again.. thanks for all your help

User avatar
joebone
Posts: 175
Joined: Sep 05 2018
Has thanked: 53 times
Been thanked: 4 times

Feb 28 2019

Here is a possible solution to this problem

I haven't tested this yet... so its unconfirmed
command.rld
https://www.multicharts.com/trading-sof ... ommandLine

Zheka
Posts: 225
Joined: Jan 13 2016
Has thanked: 8 times
Been thanked: 53 times

Mar 01 2019

This will not work reliably all the time. I could not make 'data reloading' work well in live trading..

User avatar
joebone
Posts: 175
Joined: Sep 05 2018
Has thanked: 53 times
Been thanked: 4 times

Mar 01 2019

This will not work reliably all the time. I could not make 'data reloading' work well in live trading..

just when i had hope... what was the problem?