Mid-point price for FX and 5-sec True RT bars from IB  [SOLVED]

Questions about MultiCharts and user contributed studies.
Zheka
Posts: 92
Joined: 13 Jan 2016
Has thanked: 3 times
Been thanked: 14 times

Mid-point price for FX and 5-sec True RT bars from IB

Postby Zheka » 13 Jan 2016

With this post i would like to raise the importance of this simple-to-implement functionality(ies). Simple - but critical!

As there is no "trade" data in FX (only bid and ask), one has to manually calculate a mid-point to feed into signals.
This leads to some subtle issues in RT (series syncing and waiting for the "opening tick") which are poorly documented and quite difficult to address and which delay placement of orders in RT (slippage with market orders and being late in the limit order queue).

At the moment, for FX, MC wrongly feeds Ask prices into Trade, and leaves Ask field empty(?). While IB - for years - has been streaming Mid-point price through their API.
Why does one have to waste time coding around with IOG/RecalcLastBarAfter/etc (with no result/support assured) for a 1-min-based chart? It is only logical to call things their names and use as intended, isn't it?
I have seen this topic in PM and earlier filed a PM request myself. But these are "under review", etc...

Another topic is about matching "Historical" to "RT" (snapshot) data from IB.
Besides slight differences in data, a problem arises in RT when you - for some reason - re-load a chart and it backfills differently, leading to a different strategy position. And then you have to get into syncing, etc,etc...

Unchecking "download missing historical data" does not seem to do good.
I found that when using RT bid and ask prices, indicator readings are volatile and may lead to generation of a signal/order erroneously, and when you reload a chart, they become smooth, as intended..(from experience, using 'real-time history matching' does not solve the problem but may lead to delay in order placements).

I tried to work around by programmatically reloading an x-min chart every 3-4 bars (which makes MC retrieve "historical" bars from IB). Despite some unpredictable behavior (script recalculations strange number of times) and need to make some changes to code (order placement at recalc, turning auto-trading on,etc), this works 99% of the time.
Complications arise when you trade several time-frames on an instrument and with IB pacing violations as the quantity of instruments grows past 4-5. And then there are cases when such set up crashes either IB Gateway or MC or leads to memory overflow. One cannot really rely upon such set up for automatic trading.

To address this problem IB stream "True Historical 5-sec bars" via their API, with no delay, rock solid. No need to pay for 3rd party data for the majority of users.

Would be really great to see this functionality implemented.
I understand the limitation of storing either 1-tick or 1min bars in the database. The easiest would be probably to store only 1-min bars built from such 5-sec bars in memory.
These users thanked the author Zheka for the post:
Grigorios

AlphaCat
Posts: 69
Joined: 09 Jul 2013
Been thanked: 9 times

Re: Mid-point price for FX and 5-sec True RT bars from IB

Postby AlphaCat » 16 Jan 2016

I agree that the bid/ask midpoint should generally be used as trade tick, not only for IB. Its just the logical choice, instead of using either bid or ask. Especially when spreads widen, or less liquid pairs, the midpoint is the most representatice way to chart. Some of the EM crosses have really wide bid/ask spread.

Having the option to use bid/ask midpoint is not only useful for fx, but also for instruments that trade infequently and thus produce few trade ticks, such as for example deferred futures contracts or deep otm options.
These users thanked the author AlphaCat for the post:
Grigorios

User avatar
Henry MultiСharts
Posts: 8511
Joined: 25 Aug 2011
Has thanked: 1213 times
Been thanked: 2733 times

Re: Mid-point price for FX and 5-sec True RT bars from IB

Postby Henry MultiСharts » 25 Jan 2016

Dear users,

We are going to add support for the MidPoint values from IB data feed in the next minor platform update.
As for the 5 second bars - please refer to the previous discussion on the matter.
These users thanked the author Henry MultiСharts for the post:
mickatrade

AlphaCat
Posts: 69
Joined: 09 Jul 2013
Been thanked: 9 times

Re: Mid-point price for FX and 5-sec True RT bars from IB

Postby AlphaCat » 25 Jan 2016

Could you do the same for lmax?
These users thanked the author AlphaCat for the post:
Grigorios

Zheka
Posts: 92
Joined: 13 Jan 2016
Has thanked: 3 times
Been thanked: 14 times

Re: Mid-point price for FX and 5-sec True RT bars from IB

Postby Zheka » 25 Jan 2016

Great news on coming availability of mid-point data.

As for 5-sec True bars:

We have tested IB 5 second bars. This is a realtime data snapshot coming once in 5 seconds.
How do you suggest using it? Do you want to receive a realtime tick bar once in 5 seconds instead of every second? We do not think that is a good idea.

Do you want to merge 1 second bars and 5 seconds bars to form a single data series? Or 5 second bars should filter the 1 second bars? This is not a trivial task and there is no obvious algorithm for such functionality.


I can see at least 3 significant advantages of using True RT 5-sec bars:
1. "What you trade is what you tested on".
- you do want to test on clean historical data and you do wish to have RT as close as possible to such "clean" historical data FROM THE SAME SOURCE, especially for FX.
- MC does not store RT data and reloads them with historical bars. So, there are discrepancies between trades done in RT/ positions and those done after reloading a chart (for post trade verification; after opening a new chart with the same instrument, after closing MC, etc)

2. 5-sec True bars come appr.300ms after the time-based bar close. Guaranteed. So, in principle MC can reliably "close a bar" without waiting for the opening tick of the new bar to do signal calculations and fire orders. All without using IOG/RecalcAfter, and related coding CPU/MC load (critical if using complex indicators and many charts) which lead to delays in order placement.

3. Avoiding IB's "Pacing violations" (if trading from decent number of open charts).

I can think of the following implementations in MC:
1. Build/store 1-min bars from True 5-sec bars in RT by default.
- for mid-point/bid/ask
- close the bar on arrival of the last 5-sec true bar
This should satisfy most practical uses of such functionality with MC (i.e. operating at 1-min+)
and/or
2. Add the option to "store 5-sec bars as ticks".
- ideally - on a symbol level
- if done on a "data feed level" - need to have a "smart" algo to create range bars from 5-sec
- volume - ?

3. Introduce additional storage unit of 5-sec for IB datafeed.
- will give max flexibility, is clean,but will probably take more effort to implement.

Grigorios
Posts: 48
Joined: 30 Nov 2007
Has thanked: 12 times
Been thanked: 4 times

Re: Mid-point price for FX and 5-sec True RT bars from IB

Postby Grigorios » 25 Jan 2016

AlphaCat wrote:Could you do the same for lmax?


i second that...

wzero
Posts: 142
Joined: 10 Jul 2014
Has thanked: 12 times
Been thanked: 10 times

Re: Mid-point price for FX and 5-sec True RT bars from IB

Postby wzero » 19 May 2017

5-sec realtime data is way too slow for HFT/scalper. HFT may only keep position for few seconds. Can this be an option in the IB data source settting to revert to MC 9 style forex data?
These users thanked the author wzero for the post:
Grigorios

User avatar
Henry MultiСharts
Posts: 8511
Joined: 25 Aug 2011
Has thanked: 1213 times
Been thanked: 2733 times

Re: Mid-point price for FX and 5-sec True RT bars from IB

Postby Henry MultiСharts » 22 May 2017

wzero wrote:5-sec realtime data is way too slow for HFT/scalper. HFT may only keep position for few seconds. Can this be an option in the IB data source settting to revert to MC 9 style forex data?
Hello wzero,

Previously Ask data was plotted when Trade "Quote Field" was selected. You can select Ask or Bid "Quote Field" now in Format-> Instrument-> Settings.

wzero
Posts: 142
Joined: 10 Jul 2014
Has thanked: 12 times
Been thanked: 10 times

Re: Mid-point price for FX and 5-sec True RT bars from IB

Postby wzero » 22 May 2017

Henry MultiСharts wrote:
wzero wrote:5-sec realtime data is way too slow for HFT/scalper. HFT may only keep position for few seconds. Can this be an option in the IB data source settting to revert to MC 9 style forex data?
Hello wzero,

Previously Ask data was plotted when Trade "Quote Field" was selected. You can select Ask or Bid "Quote Field" now in Format-> Instrument-> Settings.


I know that, but the charts are linked and I still want Trade quote field for futures and stocks charts. Becuase the "Quote Field" is not linked I can only set Trade quote field for all instruments type.

allenlowe
Posts: 37
Joined: 29 Jun 2011
Has thanked: 2 times
Been thanked: 3 times

Re: Mid-point price for FX and 5-sec True RT bars from IB

Postby allenlowe » 26 May 2017

Hi Henry,

I still don't get how to use the 5-sec RT. Is it for FX only? My future realtime with IB still not same as when I reload it. A 5 sec reload is good enough to me provide that the stream still update in sec. However, is it automatic to use this feature or I need to do some setting?

Thanks a lot.

Zheka
Posts: 92
Joined: 13 Jan 2016
Has thanked: 3 times
Been thanked: 14 times

Re: Mid-point price for FX and 5-sec True RT bars from IB

Postby Zheka » 01 Jun 2017

Hi Henry,

I would like to reconfirm the following re 5-sec True RT bars:

1. Minute resolution:
- IB API streams "bar updates" (i.e. O-H-L-C values) every 5-sec. How does MC update the bar in RT?

My observation is that MC updates "Close only", and that 1-min bars received in RT and after subsequent re-downloading are different.
This defeats the whole purpose of "TRUE Real-time" - which was the starting point for the request to introduce this functionality.

2. Tick resolution:
- MC updates "close only". However, switching to 5-sec bars and re-downloading historically would reveal that there are 1-4 ticks per 1 sec !
What are these ticks?

Thank you.
E.

User avatar
Henry MultiСharts
Posts: 8511
Joined: 25 Aug 2011
Has thanked: 1213 times
Been thanked: 2733 times

Re: Mid-point price for FX and 5-sec True RT bars from IB

Postby Henry MultiСharts » 01 Jun 2017

wzero wrote:
Henry MultiСharts wrote:
wzero wrote:5-sec realtime data is way too slow for HFT/scalper. HFT may only keep position for few seconds. Can this be an option in the IB data source settting to revert to MC 9 style forex data?
Hello wzero,

Previously Ask data was plotted when Trade "Quote Field" was selected. You can select Ask or Bid "Quote Field" now in Format-> Instrument-> Settings.


I know that, but the charts are linked and I still want Trade quote field for futures and stocks charts. Becuase the "Quote Field" is not linked I can only set Trade quote field for all instruments type.
We will have an option to do that in one of the future versions.
These users thanked the author Henry MultiСharts for the post:
wzero

Zheka
Posts: 92
Joined: 13 Jan 2016
Has thanked: 3 times
Been thanked: 14 times

Re: Mid-point price for FX and 5-sec True RT bars from IB

Postby Zheka » 09 Jun 2017

Dear MC,

Would greatly appreciate answers to my questions above.

Also, would be great to have this feature working for CFDs. Trade field is just empty at the moment.

Thank you.

User avatar
Henry MultiСharts
Posts: 8511
Joined: 25 Aug 2011
Has thanked: 1213 times
Been thanked: 2733 times

Re: Mid-point price for FX and 5-sec True RT bars from IB

Postby Henry MultiСharts » 09 Jun 2017

allenlowe wrote:Hi Henry, I still don't get how to use the 5-sec RT. Is it for FX only? My future realtime with IB still not same as when I reload it. A 5 sec reload is good enough to me provide that the stream still update in sec. However, is it automatic to use this feature or I need to do some setting? Thanks a lot.
Zheka wrote:Hi Henry,
I would like to reconfirm the following re 5-sec True RT bars:
1. Minute resolution:
- IB API streams "bar updates" (i.e. O-H-L-C values) every 5-sec. How does MC update the bar in RT?
My observation is that MC updates "Close only", and that 1-min bars received in RT and after subsequent re-downloading are different.
This defeats the whole purpose of "TRUE Real-time" - which was the starting point for the request to introduce this functionality.
2. Tick resolution:
- MC updates "close only". However, switching to 5-sec bars and re-downloading historically would reveal that there are 1-4 ticks per 1 sec !
What are these ticks? Thank you. E.
allenlowe, Zheka,

I believe this reply will cover both of your posts:
Midpoint data is received for Forex and Commodities only, when Trade QuoteField is selected. It does not affect the other security types loaded from IB.
Historical Midpoint bars from IB are complete OHLC bars. A tick is generated on each change of OHLC price. That is possible to choose 1 sec or 5 sec bars for historical data in IB data feed settings:
Plot ticks either from 1 second snapshots or 5 second snapshots from server.
Realtime bars are updated once in 5 seconds with a single Close value. This update does not work as actual correction, bar’s OHL values will not be changed, only the close will be updated. Realtime bars can be different from historical bars, that is expected behavior.

Zheka wrote:Dear MC,
Would greatly appreciate answers to my questions above.
Also, would be great to have this feature working for CFDs. Trade field is just empty at the moment.
Thank you.
CFDs from IB are working the following way:

Stock CFD – historical and real-time data for Ask, Bid and Trade is available.
Index CFD – historical and real-time data for Ask and Bid is available.

Zheka
Posts: 92
Joined: 13 Jan 2016
Has thanked: 3 times
Been thanked: 14 times

Re: Mid-point price for FX and 5-sec True RT bars from IB

Postby Zheka » 09 Jun 2017

Realtime bars are updated once in 5 seconds with a single Close value. This update does not work as actual correction, bar’s OHL values will not be changed, only the close will be updated. Realtime bars can be different from historical bars, that is expected behavior.

IB streams 5-sec True RT bars from their Historical data servers. Therefore, Realtime (with a 250-300ms delay) = Historical.
The whole point of this feature is exactly in making sure there is no difference between Real-time and Historical!
Therefore, it is very important to update all 1min bar O-H-L-C values with each 5-sec update.
Updating 'close only" might be an "expected behavior" at the moment, but not a practically useful one.
Moreover, bars built in RT in such way are different from both received from a RT server and from a Historical one after redownload. And H-L values might be missed completely in a faster moving market (around news releases)!
Historical Midpoint bars from IB are complete OHLC bars. A tick is generated on each change of OHLC price.

Sorry, it is not clear. A tick - in a historical bar?

If you observe 1-tick Trade data in RT - you can see 1 tick each 5-sec. However, re-downloading 1-tick data historically would reveal that there are 1-4 ticks per 1 sec !
If a piece of data is received each 5 SEC - where do these 1-4 ticks PER 1 SEC come from?

Plot ticks either from 1 second snapshots or 5 second snapshots from server


How would this work? What will be recorded in the database for tick data?

I use 5-sec True RT data from IB with SC in production, but now need to trade a complex portfolio strategy and I really hoped to implement it in PT, without having to spend lots of time coding it in C++.

User avatar
Henry MultiСharts
Posts: 8511
Joined: 25 Aug 2011
Has thanked: 1213 times
Been thanked: 2733 times

Re: Mid-point price for FX and 5-sec True RT bars from IB

Postby Henry MultiСharts » 15 Jun 2017

Zheka wrote:
Realtime bars are updated once in 5 seconds with a single Close value. This update does not work as actual correction, bar’s OHL values will not be changed, only the close will be updated. Realtime bars can be different from historical bars, that is expected behavior.
IB streams 5-sec True RT bars from their Historical data servers. Therefore, Realtime (with a 250-300ms delay) = Historical.
The whole point of this feature is exactly in making sure there is no difference between Real-time and Historical!
Therefore, it is very important to update all 1min bar O-H-L-C values with each 5-sec update.
Updating 'close only" might be an "expected behavior" at the moment, but not a practically useful one.
Moreover, bars built in RT in such way are different from both received from a RT server and from a Historical one after redownload. And H-L values might be missed completely in a faster moving market (around news releases)!
We will seek for possible improvements in one of the future versions.

Zheka wrote:
Historical Midpoint bars from IB are complete OHLC bars. A tick is generated on each change of OHLC price.
Sorry, it is not clear. A tick - in a historical bar? If you observe 1-tick Trade data in RT - you can see 1 tick each 5-sec. However, re-downloading 1-tick data historically would reveal that there are 1-4 ticks per 1 sec ! If a piece of data is received each 5 SEC - where do these 1-4 ticks PER 1 SEC come from?.
Zheka wrote:
Plot ticks either from 1 second snapshots or 5 second snapshots from server
How would this work? What will be recorded in the database for tick data?
Historical bar is split into multiple ticks. A tick is generated by MultiCharts on each change of O-H-L-C price of a historical bar. This is applicable to both 1 sec and 5 sec historical bars and this way the data is saved to the database.

Zheka
Posts: 92
Joined: 13 Jan 2016
Has thanked: 3 times
Been thanked: 14 times

Re: Mid-point price for FX and 5-sec True RT bars from IB

Postby Zheka » 19 Jun 2017

We will seek for possible improvements in one of the future versions.

Pleeease! This will make this functionality usable.

Plot ticks either from 1 second snapshots or 5 second snapshots from server
Historical bar is split into multiple ticks. A tick is generated by MultiCharts on each change of O-H-L-C price of a historical bar. This is applicable to both 1 sec and 5 sec historical bars and this way the data is saved to the database.

Switching QM to "Plot ticks from 5sec snapshots" did indeed made MC record O_H_L_C values of 5-sec bars as ticks.
This worked for "historical" bars, but not for RT data (which only had 5-sec closes recorded).

Again, "5-sec True RT data" should only be received from Historical IB servers.

So, in RT every 5-sec there will be 1-4 ticks recorded in the tick field, and 1-min bar updated with the new -O-H-L-C values.

User avatar
Henry MultiСharts
Posts: 8511
Joined: 25 Aug 2011
Has thanked: 1213 times
Been thanked: 2733 times

Re: Mid-point price for FX and 5-sec True RT bars from IB  [SOLVED]

Postby Henry MultiСharts » 13 Jul 2017

Zheka wrote:
We will seek for possible improvements in one of the future versions.

Pleeease! This will make this functionality usable.

Plot ticks either from 1 second snapshots or 5 second snapshots from server
Historical bar is split into multiple ticks. A tick is generated by MultiCharts on each change of O-H-L-C price of a historical bar. This is applicable to both 1 sec and 5 sec historical bars and this way the data is saved to the database.

Switching QM to "Plot ticks from 5sec snapshots" did indeed made MC record O_H_L_C values of 5-sec bars as ticks.
This worked for "historical" bars, but not for RT data (which only had 5-sec closes recorded).

Again, "5-sec True RT data" should only be received from Historical IB servers.

So, in RT every 5-sec there will be 1-4 ticks recorded in the tick field, and 1-min bar updated with the new -O-H-L-C values.
This improvement has been implemented in MultiCharts 10.0 Release 8 and MultiCharts 11 Beta 1.

Zheka
Posts: 92
Joined: 13 Jan 2016
Has thanked: 3 times
Been thanked: 14 times

Re: Mid-point price for FX and 5-sec True RT bars from IB

Postby Zheka » 13 Jul 2017

I have tested it, and it works as it should!

Thanks!
These users thanked the author Zheka for the post:
Henry MultiСharts


Return to “MultiCharts”