Failure to save minute data after 12 months is stored

Questions about MultiCharts and user contributed studies.
janus
Posts: 835
Joined: 25 May 2009
Has thanked: 63 times
Been thanked: 104 times

Failure to save minute data after 12 months is stored

Postby janus » 01 Jan 2010

I need confirmation of a possible serious data collection and storing problem with 5.5 for minute data using IB. This applies to indexes. I use INDU (NYSE) and SPI (SNFE). Now I realise IB only offers 12 months of intraday data. However, when I try to backfill say 16 months of 1 minute data for one of the symbols (2 months at a time to avoid the annoying pacing violation) something gets corrupted and it prevents me from backfilling the other symbol in the last 12 months. What's worse is real-time minute data is collected and displayed fine in MC but is not stored into the database for either symbol. Tick and daily data are fine. The only way I have found to fix this is to delete the database files, run the installer to repair the installation, reboot, add back the symbols. Simply deleting the symbols using QM and adding the symbols back with no data doesn't help.

I noticed there was a problem after I upgraded from 5.0 to 5.5. Before the upgrade I already had more than 12 months of minute data for both INDU and SPI. After the upgrade the real-time minute data stopped being stored in the database although I could plot it in real-time (of course each time I do a reload or change resolution the data displayed is lost). I replicated the problem on another PC using a brand new install of 5.5 (running Vista instead of XP).

I suspect any user who collects minute data will find MC will stop storing it in the database after more than 12 or so months of data is saved. I've reported it to support but want to know if others can see the same problem in case it's just me, or if anyone has any ideas. I haven't tried it on shares nor futures (hard to get futures to go back that far anyway).

User avatar
TJ
Posts: 7740
Joined: 29 Aug 2006
Location: Global Citizen
Has thanked: 1033 times
Been thanked: 2221 times

Postby TJ » 01 Jan 2010

I observed the same behavior... but wasn't able to pin point the problem.

Your description seems like what is happening.



p.s. Is there anybody out there with an intact historical minute database for SPX?
mine has been butchered through all these errors.

tekram
Posts: 96
Joined: 26 May 2009
Has thanked: 6 times
Been thanked: 18 times

Postby tekram » 01 Jan 2010

This doesn't seem to be a problem with MC6 beta, with data originally from TS.

SPX text export 1/16/2007 to 12/31/2009 compressed to rar. Forum limitation at 2 MB.
Attachments
SPX_TS1min20y.part01.rar
SPX exported pt 1
(1.39 MiB) Downloaded 132 times
SPX_TS1min20y.part02.rar
SPX exported pt 2
(780.4 KiB) Downloaded 133 times

janus
Posts: 835
Joined: 25 May 2009
Has thanked: 63 times
Been thanked: 104 times

Postby janus » 02 Jan 2010

TJ, that's both good and bad news. Good in that it confirms it's a real problem and not just me, bad in that it's a very serious problem, and so must be fixed before 6.0 is officially released.

brodnicki steven
Posts: 407
Joined: 01 Jan 2008
Been thanked: 3 times

Postby brodnicki steven » 02 Jan 2010

Not sure if this is the same problem or not but when I import say 3 years of spx ascii data and then "merge data" with spx real-time, the result is about 1 year of spx data. It happens everytime I try it. Seems to be a 1 year limit for some reason.

janus
Posts: 835
Joined: 25 May 2009
Has thanked: 63 times
Been thanked: 104 times

Postby janus » 02 Jan 2010

brodnicki steven, not sure what you mean by "the result is about 1 year of spx data". When I did an import of over 12 months if minute data, the real-time data was not stored in the database at all. I think I could still scroll back through the historical data - but then I could be wrong as I can't remember any more because I haven't re-imported it after doing my nth "repair install".

brodnicki steven
Posts: 407
Joined: 01 Jan 2008
Been thanked: 3 times

Postby brodnicki steven » 02 Jan 2010

brodnicki steven, not sure what you mean by "the result is about 1 year of spx data". When I did an import of over 12 months if minute data, the real-time data was not stored in the database at all. I think I could still scroll back through the historical data - but then I could be wrong as I can't remember any more because I haven't re-imported it after doing my nth "repair install".
* My problem was that it would only plot 1 year of data, regardless of how big the ascii file was. I needed all 3 years for testing. I'll have to check and see if the real time data is stored, it plots in RT from the end of the ascii data.
I haven't had to do any repair installs, MC6 beta1 and 2 are stable for me.

janus
Posts: 835
Joined: 25 May 2009
Has thanked: 63 times
Been thanked: 104 times

Postby janus » 02 Jan 2010

OK. I can't recall if I had that issue as well. I might have. I won't try again as I have everything working fine so far and I don't want to go through yet another "repair install". It better be fixed in 6.0.

janus
Posts: 835
Joined: 25 May 2009
Has thanked: 63 times
Been thanked: 104 times

Postby janus » 04 Jan 2010

Now the problem has got worse. Even though I don't have anywhere near 12 months of data for any of the indexes, real-time display works but no new data is stored in the database. So, each time I do a reload or exit/start MC, I loose all the historical data over the past few days.

janus
Posts: 835
Joined: 25 May 2009
Has thanked: 63 times
Been thanked: 104 times

Postby janus » 05 Jan 2010

I checked out the settings for the Data Source

QM -> Tools -> Data Sources -> Interactive Brokers -> Settings

and noticed "Filter bars with zero volume was ticked" So I unticked it, reloaded the two indexes and the historical data finally was retrieved and stored. It doesn't explain why data was successfully saved up to a few days ago then stopped. This is not consistent. So, what's the story? I would prefer to leave the setting on to filter bars with zero volume for futures. IB doesn't appear to include volume for indexes so by default MC should ignore that setting, and probably did in prior versions. So, soemthing odd is happening.

futurespeter
Posts: 2
Joined: 06 Jan 2010

Indices minute quotes not stored

Postby futurespeter » 28 Jan 2010

I’ve same problem, using Version 5.5 Release (Build 2723) together with IB.

I get realtime quotes INDU and DTX, but MC does only store daily- and tick-quotes, no minute-quotes are stored.

Please, what is going wrong here?

ciao

futurespeter

janus
Posts: 835
Joined: 25 May 2009
Has thanked: 63 times
Been thanked: 104 times

Re: Indices minute quotes not stored

Postby janus » 28 Jan 2010

It's a bug - fixed in 6.0 beta according to MC. Should have a bug fix for 5.5 but I'm prepared to wait for the official release of 6.0, which is reported to be soon.

futurespeter
Posts: 2
Joined: 06 Jan 2010

Postby futurespeter » 28 Jan 2010

thanks a lot for information.

Spaceant
Posts: 254
Joined: 30 May 2009
Has thanked: 1 time
Been thanked: 3 times

Re: Indices minute quotes not stored

Postby Spaceant » 28 Jan 2010

It's a bug - fixed in 6.0 beta according to MC. Should have a bug fix for 5.5 but I'm prepared to wait for the official release of 6.0, which is reported to be soon.
Hi,

I have experieced this as well after I updated from MC 5.0 to MC 5.5. The min data of HSI, with more than 1 year data, doesn't store after installation. The strange thing is that it can update on chart during trading hours, but it doesn't store on the database. I have called up the live support, they have copied three files on my computer and did something that probrably upated the "tsIBDataFeed.DLL". The problem have been fixed. I told to the liveTSSupport team and requested them to post the files to let others to fix the problem. She said she would talk to the management.

If the files have not been posted up, you can call TSSupport to fix the problem, no need to wait for MC6.0 for a fix!!

Sa
Last edited by Spaceant on 28 Jan 2010, edited 1 time in total.

User avatar
TJ
Posts: 7740
Joined: 29 Aug 2006
Location: Global Citizen
Has thanked: 1033 times
Been thanked: 2221 times

Postby TJ » 28 Jan 2010

this experience highlights the need for a one-button backup and restore.

every professional grade database should have one.

Spaceant
Posts: 254
Joined: 30 May 2009
Has thanked: 1 time
Been thanked: 3 times

Postby Spaceant » 28 Jan 2010

this experience highlights the need for a one-button backup and restore.

every professional grade database should have one.
What do you mean by one-button backup and restore? Are you asking TSSupport to prepare something to let users use to do something that enable them to backup or restore by pressing one key (or tool)..... or are you asking users to well prepared before each upgrade, by backing up database in case someting bad happens...

Sa

User avatar
TJ
Posts: 7740
Joined: 29 Aug 2006
Location: Global Citizen
Has thanked: 1033 times
Been thanked: 2221 times

Postby TJ » 28 Jan 2010

this experience highlights the need for a one-button backup and restore.

every professional grade database should have one.
What do you mean by one-button backup and restore? Are you asking TSSupport to prepare something to let users use to do something that enable them to backup or restore by pressing one key (or tool).

Sa

instead of looking for the location of the database... and the gymnastics of copying and pasting for backup...

press ONE BUTTON and your database is backed up...

press another button and your archived database is restored.


if your data base was corrupted, simply go to one of the archived version and make a restoration.

Spaceant
Posts: 254
Joined: 30 May 2009
Has thanked: 1 time
Been thanked: 3 times

Postby Spaceant » 28 Jan 2010

Just share what have been doing for "backup and restore". I use a third pary software - SyncBack which synchronizes files, folders or even disks. I use it to synchronize the database folders and the program folder to another disk on the same computer (in addidtion, I used to synchronize them to another removable harddisk as well, but I have been lazy these days!!), a protection plus a protection.

I do the sychronization evey few days, by making a few clicks. Then I have the total backups. Whenever there is any corruption on database file or program files, I can reverse the condition back to the last synchronization. This is the maintenance routine I have done and will certainly keep on doing. Also, it is not a real headache to have my computer re-installation, as I would always do the sychronization step before it and all I need to do i to do the reverse by clicking a few clicks.

BTY, there should be a lot of softwares available that can do the job, do a search on download dot com on backup and restore if you like.

Sa

User avatar
TJ
Posts: 7740
Joined: 29 Aug 2006
Location: Global Citizen
Has thanked: 1033 times
Been thanked: 2221 times

Postby TJ » 28 Jan 2010

...
BTY, there should be a lot of softwares available that can do the job, do a search on download dot com on backup and restore if you like.

Sa

for someone with computer knowledge... this is easy.

bear in mind most of the users are traders,
it is challenging enough for them to locate the database file directory,
let alone making useful backup and restores.

brodnicki steven
Posts: 407
Joined: 01 Jan 2008
Been thanked: 3 times

Postby brodnicki steven » 28 Jan 2010

...
BTY, there should be a lot of softwares available that can do the job, do a search on download dot com on backup and restore if you like.

Sa

for someone with computer knowledge... this is easy.

bear in mind most of the users are traders,
it is challenging enough for them to locate the database file directory,
let alone making useful backup and restores.
TJ: I like the idea of a 1 button backup and restore, should be easy to do as well.
I think even a simple batch file would suffice.

User avatar
TJ
Posts: 7740
Joined: 29 Aug 2006
Location: Global Citizen
Has thanked: 1033 times
Been thanked: 2221 times

Postby TJ » 28 Jan 2010

...
BTY, there should be a lot of softwares available that can do the job, do a search on download dot com on backup and restore if you like.

Sa

for someone with computer knowledge... this is easy.

bear in mind most of the users are traders,
it is challenging enough for them to locate the database file directory,
let alone making useful backup and restores.
TJ: I like the idea of a 1 button backup and restore, should be easy to do as well.
I think even a simple batch file would suffice.

yes, i think anything is simple for a computer aware person.

talk to a trader...
i bet you 99 dollars that 99% of them do not know:
1. what the heck is a batch file
2. where is the batch file located
3. what does a batch file do, and how does it work ?
4. how to create a batch file, how to test it?
5. this brings us back to the original challenge: what the heck do we back up ???? what files? where is the database ???? am i backing up the indicators as well? what about wsp?

brodnicki steven
Posts: 407
Joined: 01 Jan 2008
Been thanked: 3 times

Postby brodnicki steven » 29 Jan 2010


for someone with computer knowledge... this is easy.

bear in mind most of the users are traders,
it is challenging enough for them to locate the database file directory,
let alone making useful backup and restores.
TJ: I like the idea of a 1 button backup and restore, should be easy to do as well.
I think even a simple batch file would suffice.

yes, i think anything is simple for a computer aware person.

talk to a trader...
i bet you 99 dollars that 99% of them do not know:
1. what the heck is a batch file
2. where is the batch file located
3. what does a batch file do, and how does it work ?
4. how to create a batch file, how to test it?
5. this brings us back to the original challenge: what the heck do we back up ???? what files? where is the database ???? am i backing up the indicators as well? what about wsp?
TJ: Yes, I agree, most people here are TRADERS, NOT Programmers. I think MC support sometimes forgets that.
File management duties need to be fast and easy for non-programmers.
I really like the push button file backup, file restore idea. (let programmers connect the batch files to the buttons)

Spaceant
Posts: 254
Joined: 30 May 2009
Has thanked: 1 time
Been thanked: 3 times

Postby Spaceant » 29 Jan 2010

TJ and Sptrader,

I agree with you too.
Sa

janus
Posts: 835
Joined: 25 May 2009
Has thanked: 63 times
Been thanked: 104 times

Re: Indices minute quotes not stored

Postby janus » 03 Feb 2010


If the files have not been posted up, you can call TSSupport to fix the problem, no need to wait for MC6.0 for a fix!!

Sa
I missed this one. MC should have a download section on the web site for bug fixes/patches such as these. One shouldn't have to wait for the next version to have such a serious problem fixed.

Spaceant
Posts: 254
Joined: 30 May 2009
Has thanked: 1 time
Been thanked: 3 times

Re: Indices minute quotes not stored

Postby Spaceant » 03 Feb 2010


If the files have not been posted up, you can call TSSupport to fix the problem, no need to wait for MC6.0 for a fix!!

Sa
I missed this one. MC should have a download section on the web site for bug fixes/patches such as these. One shouldn't have to wait for the next version to have such a serious problem fixed.
That's what I suggested to the live help team. Call them now!

Sa

Nick
Posts: 496
Joined: 04 Aug 2006
Has thanked: 4 times
Been thanked: 24 times

Postby Nick » 03 Feb 2010

this experience highlights the need for a one-button backup and restore.

every professional grade database should have one.
Been asking for better data base tools for years. It perplexes me that TSS seem to tick boxes rather than provide robust core functionality.


Return to “MultiCharts”