Failure to save minute data after 12 months is stored
Failure to save minute data after 12 months is stored
I need confirmation of a possible serious data collection and storing problem with 5.5 for minute data using IB. This applies to indexes. I use INDU (NYSE) and SPI (SNFE). Now I realise IB only offers 12 months of intraday data. However, when I try to backfill say 16 months of 1 minute data for one of the symbols (2 months at a time to avoid the annoying pacing violation) something gets corrupted and it prevents me from backfilling the other symbol in the last 12 months. What's worse is real-time minute data is collected and displayed fine in MC but is not stored into the database for either symbol. Tick and daily data are fine. The only way I have found to fix this is to delete the database files, run the installer to repair the installation, reboot, add back the symbols. Simply deleting the symbols using QM and adding the symbols back with no data doesn't help.
I noticed there was a problem after I upgraded from 5.0 to 5.5. Before the upgrade I already had more than 12 months of minute data for both INDU and SPI. After the upgrade the real-time minute data stopped being stored in the database although I could plot it in real-time (of course each time I do a reload or change resolution the data displayed is lost). I replicated the problem on another PC using a brand new install of 5.5 (running Vista instead of XP).
I suspect any user who collects minute data will find MC will stop storing it in the database after more than 12 or so months of data is saved. I've reported it to support but want to know if others can see the same problem in case it's just me, or if anyone has any ideas. I haven't tried it on shares nor futures (hard to get futures to go back that far anyway).
I noticed there was a problem after I upgraded from 5.0 to 5.5. Before the upgrade I already had more than 12 months of minute data for both INDU and SPI. After the upgrade the real-time minute data stopped being stored in the database although I could plot it in real-time (of course each time I do a reload or change resolution the data displayed is lost). I replicated the problem on another PC using a brand new install of 5.5 (running Vista instead of XP).
I suspect any user who collects minute data will find MC will stop storing it in the database after more than 12 or so months of data is saved. I've reported it to support but want to know if others can see the same problem in case it's just me, or if anyone has any ideas. I haven't tried it on shares nor futures (hard to get futures to go back that far anyway).
- TJ
- Posts: 7746
- Joined: 29 Aug 2006
- Location: Global Citizen
- Has thanked: 1033 times
- Been thanked: 2224 times
This doesn't seem to be a problem with MC6 beta, with data originally from TS.
SPX text export 1/16/2007 to 12/31/2009 compressed to rar. Forum limitation at 2 MB.
SPX text export 1/16/2007 to 12/31/2009 compressed to rar. Forum limitation at 2 MB.
- Attachments
-
- SPX_TS1min20y.part01.rar
- SPX exported pt 1
- (1.39 MiB) Downloaded 141 times
-
- SPX_TS1min20y.part02.rar
- SPX exported pt 2
- (780.4 KiB) Downloaded 142 times
-
- Posts: 407
- Joined: 01 Jan 2008
- Been thanked: 3 times
brodnicki steven, not sure what you mean by "the result is about 1 year of spx data". When I did an import of over 12 months if minute data, the real-time data was not stored in the database at all. I think I could still scroll back through the historical data - but then I could be wrong as I can't remember any more because I haven't re-imported it after doing my nth "repair install".
-
- Posts: 407
- Joined: 01 Jan 2008
- Been thanked: 3 times
* My problem was that it would only plot 1 year of data, regardless of how big the ascii file was. I needed all 3 years for testing. I'll have to check and see if the real time data is stored, it plots in RT from the end of the ascii data.brodnicki steven, not sure what you mean by "the result is about 1 year of spx data". When I did an import of over 12 months if minute data, the real-time data was not stored in the database at all. I think I could still scroll back through the historical data - but then I could be wrong as I can't remember any more because I haven't re-imported it after doing my nth "repair install".
I haven't had to do any repair installs, MC6 beta1 and 2 are stable for me.
I checked out the settings for the Data Source
QM -> Tools -> Data Sources -> Interactive Brokers -> Settings
and noticed "Filter bars with zero volume was ticked" So I unticked it, reloaded the two indexes and the historical data finally was retrieved and stored. It doesn't explain why data was successfully saved up to a few days ago then stopped. This is not consistent. So, what's the story? I would prefer to leave the setting on to filter bars with zero volume for futures. IB doesn't appear to include volume for indexes so by default MC should ignore that setting, and probably did in prior versions. So, soemthing odd is happening.
QM -> Tools -> Data Sources -> Interactive Brokers -> Settings
and noticed "Filter bars with zero volume was ticked" So I unticked it, reloaded the two indexes and the historical data finally was retrieved and stored. It doesn't explain why data was successfully saved up to a few days ago then stopped. This is not consistent. So, what's the story? I would prefer to leave the setting on to filter bars with zero volume for futures. IB doesn't appear to include volume for indexes so by default MC should ignore that setting, and probably did in prior versions. So, soemthing odd is happening.
-
- Posts: 2
- Joined: 06 Jan 2010
Indices minute quotes not stored
I’ve same problem, using Version 5.5 Release (Build 2723) together with IB.
I get realtime quotes INDU and DTX, but MC does only store daily- and tick-quotes, no minute-quotes are stored.
Please, what is going wrong here?
ciao
futurespeter
I get realtime quotes INDU and DTX, but MC does only store daily- and tick-quotes, no minute-quotes are stored.
Please, what is going wrong here?
ciao
futurespeter
Re: Indices minute quotes not stored
Hi,It's a bug - fixed in 6.0 beta according to MC. Should have a bug fix for 5.5 but I'm prepared to wait for the official release of 6.0, which is reported to be soon.
I have experieced this as well after I updated from MC 5.0 to MC 5.5. The min data of HSI, with more than 1 year data, doesn't store after installation. The strange thing is that it can update on chart during trading hours, but it doesn't store on the database. I have called up the live support, they have copied three files on my computer and did something that probrably upated the "tsIBDataFeed.DLL". The problem have been fixed. I told to the liveTSSupport team and requested them to post the files to let others to fix the problem. She said she would talk to the management.
If the files have not been posted up, you can call TSSupport to fix the problem, no need to wait for MC6.0 for a fix!!
Sa
Last edited by Spaceant on 28 Jan 2010, edited 1 time in total.
- TJ
- Posts: 7746
- Joined: 29 Aug 2006
- Location: Global Citizen
- Has thanked: 1033 times
- Been thanked: 2224 times
What do you mean by one-button backup and restore? Are you asking TSSupport to prepare something to let users use to do something that enable them to backup or restore by pressing one key (or tool)..... or are you asking users to well prepared before each upgrade, by backing up database in case someting bad happens...this experience highlights the need for a one-button backup and restore.
every professional grade database should have one.
Sa
- TJ
- Posts: 7746
- Joined: 29 Aug 2006
- Location: Global Citizen
- Has thanked: 1033 times
- Been thanked: 2224 times
What do you mean by one-button backup and restore? Are you asking TSSupport to prepare something to let users use to do something that enable them to backup or restore by pressing one key (or tool).this experience highlights the need for a one-button backup and restore.
every professional grade database should have one.
Sa
instead of looking for the location of the database... and the gymnastics of copying and pasting for backup...
press ONE BUTTON and your database is backed up...
press another button and your archived database is restored.
if your data base was corrupted, simply go to one of the archived version and make a restoration.
Just share what have been doing for "backup and restore". I use a third pary software - SyncBack which synchronizes files, folders or even disks. I use it to synchronize the database folders and the program folder to another disk on the same computer (in addidtion, I used to synchronize them to another removable harddisk as well, but I have been lazy these days!!), a protection plus a protection.
I do the sychronization evey few days, by making a few clicks. Then I have the total backups. Whenever there is any corruption on database file or program files, I can reverse the condition back to the last synchronization. This is the maintenance routine I have done and will certainly keep on doing. Also, it is not a real headache to have my computer re-installation, as I would always do the sychronization step before it and all I need to do i to do the reverse by clicking a few clicks.
BTY, there should be a lot of softwares available that can do the job, do a search on download dot com on backup and restore if you like.
Sa
I do the sychronization evey few days, by making a few clicks. Then I have the total backups. Whenever there is any corruption on database file or program files, I can reverse the condition back to the last synchronization. This is the maintenance routine I have done and will certainly keep on doing. Also, it is not a real headache to have my computer re-installation, as I would always do the sychronization step before it and all I need to do i to do the reverse by clicking a few clicks.
BTY, there should be a lot of softwares available that can do the job, do a search on download dot com on backup and restore if you like.
Sa
- TJ
- Posts: 7746
- Joined: 29 Aug 2006
- Location: Global Citizen
- Has thanked: 1033 times
- Been thanked: 2224 times
...
BTY, there should be a lot of softwares available that can do the job, do a search on download dot com on backup and restore if you like.
Sa
for someone with computer knowledge... this is easy.
bear in mind most of the users are traders,
it is challenging enough for them to locate the database file directory,
let alone making useful backup and restores.
-
- Posts: 407
- Joined: 01 Jan 2008
- Been thanked: 3 times
TJ: I like the idea of a 1 button backup and restore, should be easy to do as well....
BTY, there should be a lot of softwares available that can do the job, do a search on download dot com on backup and restore if you like.
Sa
for someone with computer knowledge... this is easy.
bear in mind most of the users are traders,
it is challenging enough for them to locate the database file directory,
let alone making useful backup and restores.
I think even a simple batch file would suffice.
- TJ
- Posts: 7746
- Joined: 29 Aug 2006
- Location: Global Citizen
- Has thanked: 1033 times
- Been thanked: 2224 times
TJ: I like the idea of a 1 button backup and restore, should be easy to do as well....
BTY, there should be a lot of softwares available that can do the job, do a search on download dot com on backup and restore if you like.
Sa
for someone with computer knowledge... this is easy.
bear in mind most of the users are traders,
it is challenging enough for them to locate the database file directory,
let alone making useful backup and restores.
I think even a simple batch file would suffice.
yes, i think anything is simple for a computer aware person.
talk to a trader...
i bet you 99 dollars that 99% of them do not know:
1. what the heck is a batch file
2. where is the batch file located
3. what does a batch file do, and how does it work ?
4. how to create a batch file, how to test it?
5. this brings us back to the original challenge: what the heck do we back up ???? what files? where is the database ???? am i backing up the indicators as well? what about wsp?
-
- Posts: 407
- Joined: 01 Jan 2008
- Been thanked: 3 times
TJ: Yes, I agree, most people here are TRADERS, NOT Programmers. I think MC support sometimes forgets that.TJ: I like the idea of a 1 button backup and restore, should be easy to do as well.
for someone with computer knowledge... this is easy.
bear in mind most of the users are traders,
it is challenging enough for them to locate the database file directory,
let alone making useful backup and restores.
I think even a simple batch file would suffice.
yes, i think anything is simple for a computer aware person.
talk to a trader...
i bet you 99 dollars that 99% of them do not know:
1. what the heck is a batch file
2. where is the batch file located
3. what does a batch file do, and how does it work ?
4. how to create a batch file, how to test it?
5. this brings us back to the original challenge: what the heck do we back up ???? what files? where is the database ???? am i backing up the indicators as well? what about wsp?
File management duties need to be fast and easy for non-programmers.
I really like the push button file backup, file restore idea. (let programmers connect the batch files to the buttons)
Re: Indices minute quotes not stored
I missed this one. MC should have a download section on the web site for bug fixes/patches such as these. One shouldn't have to wait for the next version to have such a serious problem fixed.
If the files have not been posted up, you can call TSSupport to fix the problem, no need to wait for MC6.0 for a fix!!
Sa
Re: Indices minute quotes not stored
That's what I suggested to the live help team. Call them now!I missed this one. MC should have a download section on the web site for bug fixes/patches such as these. One shouldn't have to wait for the next version to have such a serious problem fixed.
If the files have not been posted up, you can call TSSupport to fix the problem, no need to wait for MC6.0 for a fix!!
Sa
Sa