Better Topic: IB and its major backfill problems.

Questions about MultiCharts and user contributed studies.
bowlesj3
Posts: 2180
Joined: 21 Jul 2007
Has thanked: 227 times
Been thanked: 429 times

Better Topic: IB and its major backfill problems.

Postby bowlesj3 » 14 Oct 2008

IB has a problem with backfilling during the day their E-Mini S&P 500 data. It has been happening about 2 weeks now. It is not an MC problem but MC aborting and requiring a startup picks up the flat bars. I am curious if others are experiencing this. See the attached picture. I sent an email to tac@interactivebrokers.com to complain. Maybe others want to do the same.
Attachments
1MinBarChartProblem.jpg
(41.04 KiB) Downloaded 783 times
Last edited by bowlesj3 on 17 Oct 2008, edited 2 times in total.

bowlesj3
Posts: 2180
Joined: 21 Jul 2007
Has thanked: 227 times
Been thanked: 429 times

Postby bowlesj3 » 16 Oct 2008

Elite Trader has a discussion on this for any interested.

http://www.elitetrader.com/vb/showthrea ... to=newpost

brodnicki steven
Posts: 407
Joined: 01 Jan 2008
Been thanked: 3 times

Postby brodnicki steven » 17 Oct 2008

Elite Trader has a discussion on this for any interested.

http://www.elitetrader.com/vb/showthrea ... to=newpost

The link doesn't seem to be working. I can't get historical data from IB today either.

User avatar
TJ
Posts: 7740
Joined: 29 Aug 2006
Location: Global Citizen
Has thanked: 1033 times
Been thanked: 2221 times

Postby TJ » 17 Oct 2008

welcome to the club. :(

bowlesj3
Posts: 2180
Joined: 21 Jul 2007
Has thanked: 227 times
Been thanked: 429 times

Postby bowlesj3 » 17 Oct 2008

I just clicked the URL thread on my second posting above and it took me to the latest EliteTrader posting on that thread. There at least two threads going on about this IB flat bar problem. Last night I got up at 3am and had flat bars since 17:23 the night before until 3am. You get at least 1 hour to 2.5 hours of flat bars if you restart during the day.

Everything just locked up on me. I reset the database and the ESZ8 status is wait. No live, no backfill. nothing. However my second symbol SXFZ08 is fine on all fronts. So as long as IB stays in business in these hard economic times maybe we are okay with SXFZ08. Its a peek-a-boo symbol with no overnight trading and big gaps at times the next morning.

winsome
Posts: 13
Joined: 09 Oct 2008
Location: England

Postby winsome » 17 Oct 2008

i have been having same problems downloading data re ib / multi charts. get ym 75 tick and ym 250. but 750 backfilling forever. es i am still waiting for data after 3 hrs. says tws warning message ID=-1 market data farm is ok.

bowlesj3
Posts: 2180
Joined: 21 Jul 2007
Has thanked: 227 times
Been thanked: 429 times

Postby bowlesj3 » 17 Oct 2008

I think this is an IB and Globex problem. ESZ8 and NQZ8 which is also globex are both locked. However SXFZ08 which is the Montreal exchange is fine on IB using MC..

Someone suggested shutting off the backfill on MC but I need the history and have better things to do than wait for enough data.

winsome
Posts: 13
Joined: 09 Oct 2008
Location: England

Postby winsome » 17 Oct 2008

i allso have q tracker with ib feed. no back fill there allso, only starts charting from when log on. 2 hr period reads 3 mins now.

winsome
Posts: 13
Joined: 09 Oct 2008
Location: England

Postby winsome » 17 Oct 2008

finally got es up. 100 tick and 400 tick. must be waiting till chart filled before getting charts up

bowlesj3
Posts: 2180
Joined: 21 Jul 2007
Has thanked: 227 times
Been thanked: 429 times

Postby bowlesj3 » 17 Oct 2008

I think Harld on Elite Trader may have nailed it. Here is his post copied in here with cut and paste. His chart is a TWS chart (not included). Here is the back end of that post. http://www.elitetrader.com/vb/showthrea ... ost2129502
=======================================
All problems should occur simultaneously on TWS charting and the software used via API, otherwise they would not be backfill related.

So the following test was done with TWS directly and NT, which I use as a front end:

All Globex instruments (ES, NQ, EMD, 6A, 6B, 6C, 6E, etc.) and all Nymex instruments (CL, RB, HO, GC, SI, etc.) took over 2 minutes to load and show flat bars on TWS. See example of Crude Oil Chart below. Screenshot directly taken from TWS. Other charts for NYMEX and GLOBEX look similar.

All IDEALPRO currency pairs show flat bars this morning, loading charts and backfill is fast though. Complete charts and fast backfill is available for EUREX, ICE, NYBOT and ECBOT Agricultural.

The problem is GLOBEX and NYMEX. Does anybody know how to convince the IT department of Interactive Brokers to maintain their infrastructure in line with the needs of their customers?

I had been so far satisfied with IB, but no charts for the main futures contracts of GLOBEX and NYMEX for more than 2 weeks is simply not acceptable.
=================================

User avatar
TJ
Posts: 7740
Joined: 29 Aug 2006
Location: Global Citizen
Has thanked: 1033 times
Been thanked: 2221 times

Postby TJ » 17 Oct 2008

an officer from IB has confirmed the back fill problem.
http://elitetrader.com/vb/showthread.ph ... ost2129528
DAV
Interactive Brokers

10-17-08 03:15 PM

Recent chart issues including slow loading, backfill problems and globex data problems are all related to hardware failures which the vendor (a major, reputable vendor) is not able to correct despite many attempts. We are switching to a different hardware platform on Monday to correct the problem.

User avatar
TJ
Posts: 7740
Joined: 29 Aug 2006
Location: Global Citizen
Has thanked: 1033 times
Been thanked: 2221 times

Postby TJ » 17 Oct 2008

http://elitetrader.com/vb/showthread.ph ... ost2130033
kostia00
Interactive Brokers

10-17-08 09:58 PM

The new hardware that handles the backfill for Globex will be in place and operational by Sunday 3pm Globex open.

winsome
Posts: 13
Joined: 09 Oct 2008
Location: England

Postby winsome » 18 Oct 2008

thant you tj for your update. hope ok monday for all.

bowlesj3
Posts: 2180
Joined: 21 Jul 2007
Has thanked: 227 times
Been thanked: 429 times

Postby bowlesj3 » 18 Oct 2008

This problem should have been mentioned by IB immediately via a bulletin on TWS. I use to get lots of bulletins on TWS and I don't get any now. I guess I have to call IB about this or create a ticket. Maybe it got shut off during a new release install (something that should not be possible to shut off or should have a warning if you try to shut it off). It is a waste of traders time having to communicate via these posts to start guessing what is going on.

brodnicki steven
Posts: 407
Joined: 01 Jan 2008
Been thanked: 3 times

Postby brodnicki steven » 18 Oct 2008

That's fantastic news !!!
I was afraid that I'd have to change data feeds. I can't wait to see it work on Sunday.

bowlesj3
Posts: 2180
Joined: 21 Jul 2007
Has thanked: 227 times
Been thanked: 429 times

Postby bowlesj3 » 18 Oct 2008

I think all of us knew they would have to fix this. It is just a matter of how fast would they realize it and actually do it. I am scratching my head at the way they handled it.

User avatar
TJ
Posts: 7740
Joined: 29 Aug 2006
Location: Global Citizen
Has thanked: 1033 times
Been thanked: 2221 times

Postby TJ » 18 Oct 2008

this episode also exposed two detrimental fault of MultiCharts...

(1) When the data feed is not supplying backfill data, MultiChart should be intelligent enough to do the following:

after waiting for X number of minutes (or retries), MC should:
-- flash a pop-up window with the warning message that "Back Fill is not forthcoming".
-- Give user a choice to keep waiting, or to go on without downloading historical data.

Right now MC just sit there acting like a fool... frozen, waiting...

for the few people who are savvy enough to check the QM... might not be technical enough to understand the log.

(2) On Friday when this problem hit me, I quickly recognized the problem was in the back fill.
I immediately turned off the back fill, restarted MC, and lost only 10 minutes of my day.
However this would not be a good alternative for people who trade Globex as well as other exchanges, because you cannot selectively disable backfill to certain instruments or exchanges. Once you disable backfill, it affects all instruments/exchanges across MC.

... anyway... I think MC can do better.



p.s. come to think of it, I did see a pop up window, but it did not get focus and was hidden under all the other windows. I did not see it until the end of the day when I was shutting down the computer.
Furthermore, there is no reference that the window was from MC. The message on the window was not informative at all. I could not see the purpose of the message, nor the alternative action available to the user.
(I have attached a screen shot of the window for your reference.)
Attachments
datafeed couldnt be loaded.gif
(3.68 KiB) Downloaded 753 times

bowlesj3
Posts: 2180
Joined: 21 Jul 2007
Has thanked: 227 times
Been thanked: 429 times

Postby bowlesj3 » 18 Oct 2008

I like the idea FS. However at times even at 5am the backfill is slow but it does finally come (maybe 10 minutes or 20 minutes later). So when that popup comes up it should still be trying and be removed by MC if the backfill does come since I often leave and come back and would not want that popup to stop the trying.

Regarding multiple exchanges, I trade one symbol from Globex and one symbol from Montreal. The Montreal one came down no problem with backfill and live data. Of course the Globex one just sat their all day. Trading the Montreal one was no problem. Anyway, I agree, if you want to trade both then there should be checkboxes for each symbol so one could keep the backfill for the one symbol and not for the other. The popup for any symbols could be symbol specific also.

bowlesj3
Posts: 2180
Joined: 21 Jul 2007
Has thanked: 227 times
Been thanked: 429 times

Postby bowlesj3 » 18 Oct 2008

Regarding two symbols (one with backfill problems in one workspaces) and (another with no backfill problems in a different workspace), I am wondering if the workspace that is waiting could be closed. Then maybe one could change (file,preferences, dataserver mode) to not backfill and reopen the workspace with the struggling symbol. That should be okay since the other symbol is already backfilled properly.

winsome
Posts: 13
Joined: 09 Oct 2008
Location: England

Postby winsome » 18 Oct 2008

q tracker with ib feed seems to work a differant way. starts charts then asks for backfill so even on friday my charts built from opening without me doing anything, at least after 15 mins i had some data to work with.

brodnicki steven
Posts: 407
Joined: 01 Jan 2008
Been thanked: 3 times

Postby brodnicki steven » 18 Oct 2008

I've noticed that on some pre-made charts, the historical data won't load but if I delete the chart and build a new one it does load. Not sure what causes this to happen.
Has anyone else experienced this ?

bowlesj3
Posts: 2180
Joined: 21 Jul 2007
Has thanked: 227 times
Been thanked: 429 times

Postby bowlesj3 » 20 Oct 2008

I seem to remember having a problem with a chart and then recreating it and this solved the problem. I can't remember that problem I was having but it was not this backfill problem though. So there must be some things that are initialized when a new chart is created as opposed to doing a copy and paste of the chart.


Return to “MultiCharts”