I am curious if anyone else is experiencing this.
I have to upgrade yet to test the latest beta (touch wood this weekend) but on MC 6.0 beta 2, for some reason the offline(stand alone) backfill on the 5 minute bars through 60 minute bars is taking about 25 minutes (this morning and tonight). It has nothing to do with the machine because it is the same on the notebook and the desktop. The 1 minute bars and 10 second bars backfill just like normal. So do the daily bars. I have never seen anything like this before. Maybe something is strange at IB.
Slow Offline Backfill
Re: Slow Offline Backfill
Yes, this is a big problem if it does not clear itself today (I will find out in about an hour). It is a problem because if I want to take a snap shot of the market in bmp file format to be stored for future reference, I have to wait at least 25 to 30 minutes until those larger chart studies complete when the backfill completes..
I am thinking I can try clearing the database so I do not have all those old E-mini contacts (I never look at them anyway since my system is done). I am asking myself "how do I do that". Do I simply remove the 3 files from the directory and MC recreates them fresh if they are not there?
I am thinking I can try clearing the database so I do not have all those old E-mini contacts (I never look at them anyway since my system is done). I am asking myself "how do I do that". Do I simply remove the 3 files from the directory and MC recreates them fresh if they are not there?
Re: Slow Offline Backfill
Again, this problem only occurs during off-line (stand alone). When backfilling while connected for live trading the backfill is just as fast on the 5 minute through 60 minute bars as it is with the other bar sizes (1 minute, 10 second, daily). It does seem strange because one would think it should be the exact opposite. It is also strange because I have over 4 years never seen such a problem.
- JoshM
- Posts: 2195
- Joined: 20 May 2011
- Location: The Netherlands
- Has thanked: 1544 times
- Been thanked: 1565 times
- Contact:
Re: Slow Offline Backfill
Perhaps this thread can be of some help, in which MC support provides a way to clean up the database: viewtopic.php?f=1&t=7475&p=34030#p34030I am thinking I can try clearing the database so I do not have all those old E-mini contacts (I never look at them anyway since my system is done). I am asking myself "how do I do that". Do I simply remove the 3 files from the directory and MC recreates them fresh if they are not there?
Regards,
- Stan Bokov
- Posts: 963
- Joined: 18 Dec 2009
- Has thanked: 367 times
- Been thanked: 302 times
Re: Slow Offline Backfill
Dear bowlesj3,Again, this problem only occurs during off-line (stand alone). When backfilling while connected for live trading the backfill is just as fast on the 5 minute through 60 minute bars as it is with the other bar sizes (1 minute, 10 second, daily). It does seem strange because one would think it should be the exact opposite. It is also strange because I have over 4 years never seen such a problem.
I remember hearing about an issue with long offline backfill for ALL resolutions due to the fact that MC parses old data into sessions and that's why it took a long time. I can't recall where that was discussed at the moment, but you can try setting your sessions to 24/7 and seeing if that makes a difference. Please keep us updated on this, it seems strange to me that only one resolution would produce this result.
Re: Slow Offline Backfill
I managed to resolve this issue. To review it, during offline backfill, I was having to wait 30 minutes for backfill on the 5min, 10min, 15min, 30min and 60 minute charts while I was getting very fast (normal) backfill on the 1 minute, 10 second and daily charts. Here is what I tried and what finally worked.
1/ I tried what JoshM pointed me to which was to run the gbak.bat procedure. That did not work.
2/ I tried a repair run of MC 6.0 beta 2. That did not work.
3/ This worked. I return an old database from 30 days ago which has my current futures contract data. It was from before the problem started. It worked fine so I brought up IB's traders work station and did a backfill to fill in the missing data from that database until today. Then I closed down MC and brought it back up offline and it backfilled properly. So the conclusion was that somehow the database got corrupted (maybe when I reformatted my C drive and reinstalled everything, but I am not sure).
Thats it. So the moral of the story is to make sure you backup at least a 30 day cycle of databases so you can go back later and figure out the problem. It took me about 14 days to get around to working on this so having that 30 day history saved me from most likely having to do a fresh install of MC without all my user settings.
1/ I tried what JoshM pointed me to which was to run the gbak.bat procedure. That did not work.
2/ I tried a repair run of MC 6.0 beta 2. That did not work.
3/ This worked. I return an old database from 30 days ago which has my current futures contract data. It was from before the problem started. It worked fine so I brought up IB's traders work station and did a backfill to fill in the missing data from that database until today. Then I closed down MC and brought it back up offline and it backfilled properly. So the conclusion was that somehow the database got corrupted (maybe when I reformatted my C drive and reinstalled everything, but I am not sure).
Thats it. So the moral of the story is to make sure you backup at least a 30 day cycle of databases so you can go back later and figure out the problem. It took me about 14 days to get around to working on this so having that 30 day history saved me from most likely having to do a fresh install of MC without all my user settings.
- Stan Bokov
- Posts: 963
- Joined: 18 Dec 2009
- Has thanked: 367 times
- Been thanked: 302 times
Re: Slow Offline Backfill
Thanks for sharing bowlesj3, very nice and detailed post.I managed to resolve this issue. To review it, during offline backfill, I was having to wait 30 minutes for backfill on the 5min, 10min, 15min, 30min and 60 minute charts while I was getting very fast (normal) backfill on the 1 minute, 10 second and daily charts. Here is what I tried and what finally worked.
1/ I tried what JoshM pointed me to which was to run the gbak.bat procedure. That did not work.
2/ I tried a repair run of MC 6.0 beta 2. That did not work.
3/ This worked. I return an old database from 30 days ago which has my current futures contract data. It was from before the problem started. It worked fine so I brought up IB's traders work station and did a backfill to fill in the missing data from that database until today. Then I closed down MC and brought it back up offline and it backfilled properly. So the conclusion was that somehow the database got corrupted (maybe when I reformatted my C drive and reinstalled everything, but I am not sure).
Thats it. So the moral of the story is to make sure you backup at least a 30 day cycle of databases so you can go back later and figure out the problem. It took me about 14 days to get around to working on this so having that 30 day history saved me from most likely having to do a fresh install of MC without all my user settings.