Why do the E-mini futures daily bars not match the 60 min

Questions about MultiCharts and user contributed studies.
bowlesj3
Posts: 2180
Joined: 21 Jul 2007
Has thanked: 227 times
Been thanked: 429 times

Why do the E-mini futures daily bars not match the 60 min

Postby bowlesj3 » 16 Aug 2013

Hi,

I have somewhat recently started watching the daily bars in the SPY or E-Mini to see if there are any key bounces going on. Its not that big a deal but I have noticed that every so often there is a gap that shows up in the daily bars that really should not exist (no matter what way you slice it or dice it). In other words it is obviously some sort of bug. All you have to do is look at the attached picture and it becomes really obvious. The date of the 14th matches up perfectly in terms of the highs and lows on the two charts for the day. However it goes off on the next day (the 15th and you can see the gap in the daily bars). Maybe this should be a PM entry. I don't know. I am a bit surprised it occurs since I have noticed these gaps off and on ever since I got MC about 6 years ago in August 2007. Maybe it has something to do with the method of starting and stopping MC.

Thanks,
John
Attachments
20130816_093237_ES_60_.png
(74.51 KiB) Downloaded 1253 times

User avatar
Andrew MultiCharts
Posts: 1587
Joined: 11 Oct 2011
Has thanked: 931 times
Been thanked: 559 times

Re: Why do the E-mini futures daily bars not match the 60 mi

Postby Andrew MultiCharts » 16 Aug 2013

Hello Bowlesj3,

QuoteManager -> Tools -> Data Sources -> Interactive Brokers -> Settings -> Which boxes are not checked?

Xyzzy
Posts: 162
Joined: 19 Mar 2011
Has thanked: 43 times
Been thanked: 79 times

Re: Why do the E-mini futures daily bars not match the 60 mi

Postby Xyzzy » 16 Aug 2013

This appears to be a case of bad data from your data provider. Are you receiving your quotes from Interactive Brokers? They are notorious for having unreliable data.

I have accounts with both Interactive Brokers (for my broker) and IQFeed (for my data provider). I created a chart that shows daily bars of the e-mini using the ESU3 symbol from IB and the ESU13 symbol from IQFeed. See the attached screenshot.

Note how the IB chart (on top) has gaps, while the IQFeed chart (on bottom) does not. I think that Interactive Brokers is just generating its own unofficial "daily" bars based on the trades that occur during the regular trading session (open outcry). This is why there are gaps, since their daily bars don't include trades from the overnight session.

In contrast, the IQFeed daily bars are based on the official daily price data from the exchanges. They have the correct 24-hour high and low, and the close matches the official settlement price (rather than the "last trade" price).

There's no real solution except to switch to a better data vendor. One workaround is to create a minute-based bar using IB data for X number of minutes, where X equals the number of minutes in the combined trading session. (You'll need to calculate this.) This will approximate the "official" daily bars, but it won't match them exactly.
Attachments
ESU3 versus ESU13.gif
(44.08 KiB) Downloaded 1252 times

bowlesj3
Posts: 2180
Joined: 21 Jul 2007
Has thanked: 227 times
Been thanked: 429 times

Re: Why do the E-mini futures daily bars not match the 60 mi

Postby bowlesj3 » 16 Aug 2013

Yes it is IB data. I gather they have 3 data sources (seconds which is tick data), (minute bars) and (daily bars). There was an MC temp support person (Bruce Devault - he was a consultant and seemed to really know his stuff) who somewhat explained why they are different but he suggested the problem actually comes from the exchange itself and thus IB would have the problem since they do not attempt to adjust the data or fix the problem. I have noticed that the daily bars will be accurate during the day then during the back-fill they get messed up. I am thinking that the data vendors may be doing exactly what you are suggesting to make the daily bars more accurate. I am thinking that the MC software itself could have a feature where it builds the daily bars using the minute bar data and it properly stores them and consolidates them so that they will be forced to be correct at all times. As a programmer I see no reason this can not be done if the will is there to make it happen (with the exception of when the power goes out maybe). However if the power goes out one could just restore the backups they keep of the data files and do another back-fill to replace it and if the programming was correct it would built the daily bars to match the minute bars exactly. As far as that goes the program could be smart enough to match overnight data or not depending on what the user selected for the minute bars. So I figure this problem continues because no one cares all that much (users don't care and therefore the MC staff have not set it as a very high priority). Its probably because most users use minute bar data or the tick data. For the most part I am in that category myself. That's why I said it is not a very big deal. Its more of an minor annoyance for me. At the same time I know the MC staff have a very high standard for their software (they would really like it to be perfect if they could) and they may want to fix this problem some day by making the fix internal to their software rather than the user having to get an external feed which is essentially doing the same thing. If it is done by MC I am sure the daily bars could be made to perfectly match the minute bars.

bowlesj3
Posts: 2180
Joined: 21 Jul 2007
Has thanked: 227 times
Been thanked: 429 times

Re: Why do the E-mini futures daily bars not match the 60 mi

Postby bowlesj3 » 17 Aug 2013

I attached an image of the check boxes Andrew. I don't really have much confidence in their impact. As far as I am concerned, MC should ultimately force the daily bars to match the 60 minute bars (any minute bars). The users however on rare occasions may want one daily bar chart to not show the overnight and another daily bar chart to show the overnight (I think Arnie wanted this). Either way MC should force the daily bars to match the minute bars and there should be no related check boxes. I also see no reason for this forced matching to not be 100% perfect (eventually). What I am saying is it is I believe it is very doable and I believe the MC team is quite capable of making this happen if the users show enough interest in it. For me I do not use the daily bars to make any overly accurate decisions so it is not a big deal if they are not perfect. For some however I think they would like them perfect.
Attachments
IB_Settup.PNG
(20.12 KiB) Downloaded 1247 times

Xyzzy
Posts: 162
Joined: 19 Mar 2011
Has thanked: 43 times
Been thanked: 79 times

Re: Why do the E-mini futures daily bars not match the 60 mi

Postby Xyzzy » 17 Aug 2013

You can create "daily" bars that match the minute bars using the technique that I described above. I.e., create a minute-based bar of X minutes long, where X equals the number of minutes in the combined session.

MultiCharts is just creating bars based on the data that it is receiving from your data vendor. Interactive Brokers is providing bad data.

bowlesj3
Posts: 2180
Joined: 21 Jul 2007
Has thanked: 227 times
Been thanked: 429 times

Re: Why do the E-mini futures daily bars not match the 60 mi

Postby bowlesj3 » 17 Aug 2013

I guess with that technique you describe it would automatically start at the correct alignment for the daily bars. If it does it should in theory be very accurate unless shutting down MC at 16:15 and doing a backfill to get the remainder of the data for that day the next day creates a problem. I will have to give it a shot to see what happens.

Xyzzy
Posts: 162
Joined: 19 Mar 2011
Has thanked: 43 times
Been thanked: 79 times

Re: Why do the E-mini futures daily bars not match the 60 mi

Postby Xyzzy » 17 Aug 2013

I guess with that technique you describe it would automatically start at the correct alignment for the daily bars.
As long as you count the number of minutes correctly, it works fine. It needs to match exactly the number of minutes in the specified session, not including any breaks/pauses between sessions.

I've used it for a similar situation, to create fake "daily" bars that match only the open-outcry session. To do that, set the session to "CME: Equity Index Futures (Open Outcry)" and the length of the bars to 465 minutes. I haven't had any problems.

For what it's worth, I've submitted a feature request to make this process more automatic. However, the existing way still works.

http://www.multicharts.com/pm/viewissue ... _no=MC-856

bowlesj3
Posts: 2180
Joined: 21 Jul 2007
Has thanked: 227 times
Been thanked: 429 times

Re: Why do the E-mini futures daily bars not match the 60 mi

Postby bowlesj3 » 18 Aug 2013

Maybe I should add a comment to your request suggesting the alignment between the 1 minute bars and the daily bars be such that (if the user chooses the option) the user has absolutely nothing to do and they match up 100% perfectly every time without fail (like I said this is doable).

I was thinking about your work around. Back when Bruce Devault was giving all the technical details as to why the 1 minute bars and the 10 second bars were at time not matching and it was in fact not an IB problem but an Exchange created problem I decided to write a script to compare charts (any two charts if I remember except it left out the daily bars). The script is at the link below.
viewtopic.php?f=5&t=7750
So along this lines I was thinking about a way to write a script to get the number of minute bars within a day and creating an output file that had the date and the number of minute bars for that date (not hard to do by any stretch). It then occurred to me that Friday will have a much lower number of minute bars and Sunday will have an even lower number of minute bars. It also occurred to me that if I add Friday and Sunday bars together it will still be lower than the number of bars during the other days. So this leads to a question. On Both Friday and Sunday, does the highest high and lowest low of the minute bars match the high and low of the equivalent daily bar? In this regard in the thread at the link above, I remember commenting that maybe some day I would write a script to compare these high and low values between a minute bar chart and a larger chart such as the daily bars. I never got around to doing that. Maybe now is the time to do that.
Last edited by bowlesj3 on 18 Aug 2013, edited 1 time in total.

bowlesj3
Posts: 2180
Joined: 21 Jul 2007
Has thanked: 227 times
Been thanked: 429 times

Re: Why do the E-mini futures daily bars not match the 60 mi

Postby bowlesj3 » 18 Aug 2013

It took a while but I found Bruce Devault's comments about minute bars versus second bars and daily bars. I updated the thread at the link above to have the correct pointer to this thread as well.
viewtopic.php?f=1&t=7273#p32055
Last edited by bowlesj3 on 18 Aug 2013, edited 1 time in total.

bowlesj3
Posts: 2180
Joined: 21 Jul 2007
Has thanked: 227 times
Been thanked: 429 times

Re: Why do the E-mini futures daily bars not match the 60 mi

Postby bowlesj3 » 18 Aug 2013

Regarding a quote from Bruce,
Some platforms do offer the choice to make minute bars from tick data and disregard the data feed's native minute data type, and this typically increases consistency in matching things up, although at a cost of lower performance because the amount of data to be processed is hugely greater to achieve the same ends.
I think what I am suggesting is daily bars be built from the minute bar data during the day (I suspect that this is currently being done anyway) but at some point the daily bars be consolidated by MC to create a single record for each bar but only after the program is sure that the daily bar matches the minute bar data perfectly. Where the daily bar problems are coming from is unimportant as far as I am concerned. As long as the user trusts the minute bar data if the daily bar data is built from the minute are date (real time from it or consolidated from it) they should always match. Only a program bug in MC would cause a mismatch.

Xyzzy
Posts: 162
Joined: 19 Mar 2011
Has thanked: 43 times
Been thanked: 79 times

Re: Why do the E-mini futures daily bars not match the 60 mi

Postby Xyzzy » 18 Aug 2013

So along this lines I was thinking about a way to write a script to get the number of minute bars within a day and creating an output file that had the date and the number of minute bars for that date (not hard to do by any stretch). It then occurred to me that Friday will have a much lower number of minute bars and Sunday will have an even lower number of minute bars. It also occurred to me that if I add Friday and Sunday bars together it will still be lower than the number of bars during the other days. So this leads to a question. On Both Friday and Sunday, does the highest high and lowest low of the minute bars match the high and low of the equivalent daily bar? In this regard in the thread at the link above, I remember commenting that maybe some day I would write a script to compare these high and low values between a minute bar chart and a larger chart such as the daily bars. I never got around to doing that. Maybe now is the time to do that.
You're making this more complicated than it needs to be. :-) You just need to calculate the number of minutes in each session, not each calendar day. The sessions for the e-mini are the following:
MON – FRI: 5:00 p.m. previous day – 4:15 p.m.; trading halt from 3:15 p.m. – 3:30 p.m.
http://www.cmegroup.com/trading/equity- ... tions.html

I.e., it should be the number of minutes between 5:00 PM and 4:15 PM, minus the 15 minutes for the trading halt. By my calculation, that should be 23 hours * 60 minutes = 1380 minutes.

To create "daily" bars, just create a chart with minute bars that are 1380 minutes long, and set the session to "CME: Equity Index Futures [E-Mini] (Combined Session)." You will get the result that you're looking for.

Xyzzy
Posts: 162
Joined: 19 Mar 2011
Has thanked: 43 times
Been thanked: 79 times

Re: Why do the E-mini futures daily bars not match the 60 mi

Postby Xyzzy » 18 Aug 2013

A correction: This doesn't quite work because the session template for "CME: Equity Index Futures [E-Mini] (Combined Session)" is defined in QuoteManager to have two sessions per day -- one for the day session, and one for the overnight session. This causes MultiCharts to plot two "daily" bars per day, one for each session.

You'll need to define a custom session in QuoteManager as follows:
Sunday 17:00 - Monday 16:15
Monday 17:00 - Tuesday 16:15
Tuesday 17:00 - Wednesday 16:15
Wednesday 17:00 - Thursday 16:15
Thursday 17:00 - Friday 16:15

Then create a minute-based bar with a length of 1395 minutes (the time from 17:00 to 16:15 the next day), and the session set to this custom session. You'll get perfect "daily" bars. (You don't need to factor in the session pause.)

bowlesj3
Posts: 2180
Joined: 21 Jul 2007
Has thanked: 227 times
Been thanked: 429 times

Re: Why do the E-mini futures daily bars not match the 60 mi

Postby bowlesj3 » 18 Aug 2013

It looks like you have it down pat. That's great. I don't think too many users would think of this let alone work it all out. If your not a programmer maybe you should be - LOL. I am laughing but I am actually serious. The irony from my perspective is I am a programmer but being a programmer I don't think the user should have to do any of this. As far as I am concerned if the MC folk are going to have multiple data feeds and some of them are causing problems with the daily bars they should make MC so it works around this automatically. In fact MC should be so good that the user never even knows any of the data feeds are lacking in one way or another. Maybe I think at a higher level than the average programmer (more like a designer - maybe more like an owner of MC - LOL). Having said that I just put it on my todo list to give your suggestions a try. It would make it less annoying when I see those gaps in the daily bars. Also it could be a long time before the MC staff get MC to do what I am suggesting. So thanks for your great ideas and work. I am sure there are other users who really appreciate your sharing it as much as I do and even more. I only use the daily bars to look for the occasional time when prices may make a very hard run up or down for a few days. Normally I use only from the 60 minute bars down to the 10 second bars. But I am a bit of a fuss-bus and it annoys me when I see the gaps in the daily bars even though it has no major effect on me.

I can add a bit to this. I use to subscribe to a datafeed and I found it was very slow. I would notce that the IB execution software was very fast but the datafeed was often from 10 seconds to 3 minutes behind. This was when I was using Qcharts. When I got MC and found it could use the IB data I thought this was just great because the charts were always just as fast as the IB execution. For me this is absolutely critical and I would rather have the occasional problem in the daily bars than have any delay whatsoever in the 10 second bars. To me MC and IB is the perfect combination (I need the charts to perfectly match the speed of the execution).

Update: I just added this thread to my database of useful MC forum threads. This database contains web links and allows me to find things fast that I have seen in this forum including any posts I have created. That way if anyone happens to have daily bar issues and I happen to spot their post and read it I can point them to this thread.

bowlesj3
Posts: 2180
Joined: 21 Jul 2007
Has thanked: 227 times
Been thanked: 429 times

Re: Why do the E-mini futures daily bars not match the 60 mi

Postby bowlesj3 » 23 Aug 2013

I think the biggest negative for the MC staff (even if most do not use the daily bars) is this problem is a bit of an eye soar for sales and is out in plain view. First impressions need to be good ones for sales to be high. So I figure MC needs to know the strengths and weaknesses of all their data-feeds A.S.A.P. after getting them on board as an input into MC and come up with ways that work around these weaknesses (at the same time being very careful not to doctor the data in a way that hurts the traders).

Xyzzy
Posts: 162
Joined: 19 Mar 2011
Has thanked: 43 times
Been thanked: 79 times

Re: Why do the E-mini futures daily bars not match the 60 mi

Postby Xyzzy » 23 Aug 2013

I don't view this as a negative. MultiCharts is simply displaying the data from your broker in an accurate manner. Incorrect data will create incorrect charts.


Return to “MultiCharts”