Slow Offline Backfill

Questions about MultiCharts and user contributed studies.
bowlesj3
Posts: 1978
Joined: 21 Jul 2007
Has thanked: 155 times
Been thanked: 395 times

Slow Offline Backfill

Postby bowlesj3 » 24 May 2011

I am curious if anyone else is experiencing this.

I have to upgrade yet to test the latest beta (touch wood this weekend) but on MC 6.0 beta 2, for some reason the offline(stand alone) backfill on the 5 minute bars through 60 minute bars is taking about 25 minutes (this morning and tonight). It has nothing to do with the machine because it is the same on the notebook and the desktop. The 1 minute bars and 10 second bars backfill just like normal. So do the daily bars. I have never seen anything like this before. Maybe something is strange at IB.

bowlesj3
Posts: 1978
Joined: 21 Jul 2007
Has thanked: 155 times
Been thanked: 395 times

Re: Slow Offline Backfill

Postby bowlesj3 » 25 May 2011

Yes, this is a big problem if it does not clear itself today (I will find out in about an hour). It is a problem because if I want to take a snap shot of the market in bmp file format to be stored for future reference, I have to wait at least 25 to 30 minutes until those larger chart studies complete when the backfill completes..

I am thinking I can try clearing the database so I do not have all those old E-mini contacts (I never look at them anyway since my system is done). I am asking myself "how do I do that". Do I simply remove the 3 files from the directory and MC recreates them fresh if they are not there?

bowlesj3
Posts: 1978
Joined: 21 Jul 2007
Has thanked: 155 times
Been thanked: 395 times

Re: Slow Offline Backfill

Postby bowlesj3 » 25 May 2011

Again, this problem only occurs during off-line (stand alone). When backfilling while connected for live trading the backfill is just as fast on the 5 minute through 60 minute bars as it is with the other bar sizes (1 minute, 10 second, daily). It does seem strange because one would think it should be the exact opposite. It is also strange because I have over 4 years never seen such a problem.

User avatar
JoshM
Posts: 2075
Joined: 20 May 2011
Location: The Netherlands
Has thanked: 1523 times
Been thanked: 1480 times
Contact:

Re: Slow Offline Backfill

Postby JoshM » 25 May 2011

bowlesj3 wrote:I am thinking I can try clearing the database so I do not have all those old E-mini contacts (I never look at them anyway since my system is done). I am asking myself "how do I do that". Do I simply remove the 3 files from the directory and MC recreates them fresh if they are not there?

Perhaps this thread can be of some help, in which MC support provides a way to clean up the database: viewtopic.php?f=1&t=7475&p=34030#p34030

Regards,

bowlesj3
Posts: 1978
Joined: 21 Jul 2007
Has thanked: 155 times
Been thanked: 395 times

Re: Slow Offline Backfill

Postby bowlesj3 » 25 May 2011

Thanks JoshM. I will check into it most likely tomorrow night.
John.

User avatar
Stan Bokov
Posts: 963
Joined: 18 Dec 2009
Has thanked: 367 times
Been thanked: 300 times

Re: Slow Offline Backfill

Postby Stan Bokov » 25 May 2011

bowlesj3 wrote:Again, this problem only occurs during off-line (stand alone). When backfilling while connected for live trading the backfill is just as fast on the 5 minute through 60 minute bars as it is with the other bar sizes (1 minute, 10 second, daily). It does seem strange because one would think it should be the exact opposite. It is also strange because I have over 4 years never seen such a problem.


Dear bowlesj3,

I remember hearing about an issue with long offline backfill for ALL resolutions due to the fact that MC parses old data into sessions and that's why it took a long time. I can't recall where that was discussed at the moment, but you can try setting your sessions to 24/7 and seeing if that makes a difference. Please keep us updated on this, it seems strange to me that only one resolution would produce this result.

bowlesj3
Posts: 1978
Joined: 21 Jul 2007
Has thanked: 155 times
Been thanked: 395 times

Re: Slow Offline Backfill

Postby bowlesj3 » 28 May 2011

I managed to resolve this issue. To review it, during offline backfill, I was having to wait 30 minutes for backfill on the 5min, 10min, 15min, 30min and 60 minute charts while I was getting very fast (normal) backfill on the 1 minute, 10 second and daily charts. Here is what I tried and what finally worked.


1/ I tried what JoshM pointed me to which was to run the gbak.bat procedure. That did not work.

2/ I tried a repair run of MC 6.0 beta 2. That did not work.

3/ This worked. I return an old database from 30 days ago which has my current futures contract data. It was from before the problem started. It worked fine so I brought up IB's traders work station and did a backfill to fill in the missing data from that database until today. Then I closed down MC and brought it back up offline and it backfilled properly. So the conclusion was that somehow the database got corrupted (maybe when I reformatted my C drive and reinstalled everything, but I am not sure).

Thats it. So the moral of the story is to make sure you backup at least a 30 day cycle of databases so you can go back later and figure out the problem. It took me about 14 days to get around to working on this so having that 30 day history saved me from most likely having to do a fresh install of MC without all my user settings.
These users thanked the author bowlesj3 for the post (total 2):
Stan BokovDave Masalov

User avatar
Stan Bokov
Posts: 963
Joined: 18 Dec 2009
Has thanked: 367 times
Been thanked: 300 times

Re: Slow Offline Backfill

Postby Stan Bokov » 31 May 2011

bowlesj3 wrote:I managed to resolve this issue. To review it, during offline backfill, I was having to wait 30 minutes for backfill on the 5min, 10min, 15min, 30min and 60 minute charts while I was getting very fast (normal) backfill on the 1 minute, 10 second and daily charts. Here is what I tried and what finally worked.


1/ I tried what JoshM pointed me to which was to run the gbak.bat procedure. That did not work.

2/ I tried a repair run of MC 6.0 beta 2. That did not work.

3/ This worked. I return an old database from 30 days ago which has my current futures contract data. It was from before the problem started. It worked fine so I brought up IB's traders work station and did a backfill to fill in the missing data from that database until today. Then I closed down MC and brought it back up offline and it backfilled properly. So the conclusion was that somehow the database got corrupted (maybe when I reformatted my C drive and reinstalled everything, but I am not sure).

Thats it. So the moral of the story is to make sure you backup at least a 30 day cycle of databases so you can go back later and figure out the problem. It took me about 14 days to get around to working on this so having that 30 day history saved me from most likely having to do a fresh install of MC without all my user settings.


Thanks for sharing bowlesj3, very nice and detailed post.

bowlesj3
Posts: 1978
Joined: 21 Jul 2007
Has thanked: 155 times
Been thanked: 395 times

Re: Slow Offline Backfill

Postby bowlesj3 » 31 May 2011

No problem Stan.
John.


Return to “MultiCharts”