As I near the 4GB limit, I find that I don't see any fields in QuoteManager to easily find out what symbols are taking storage.
Can you say how to go about this?
Ideally, you would have a simple COM interface that I can use through Visual Basic to iterate through all the symbols, query their properties, and import or export them. I've wanted that for a *long* time... It is my absolutely biggest pain point of managing data.
Also, I made the suggestion long ago that there really isn't any pressing need to keep the compressed bar data in a database. You really ought to just map that into the filesystem and then it could grow as big as the filesystem (which is a lot these days).
Comments?
Anyone else grappling with the 4GB limit?
Reaching the 4GB db limit.
- Dave Masalov
- Posts: 1712
- Joined: 16 Apr 2010
- Has thanked: 51 times
- Been thanked: 489 times
Re: Reaching the 4GB db limit.
Dear jek,
To reduce your database you need to delete symbols that you want and follow these instructions: viewtopic.php?t=7475&start=18
At the moment, the only way to see how much data you have for each symbol is to right click --> Edit Data and load the interval for which you have data.
I have forwarded your suggestions to the developers.
To reduce your database you need to delete symbols that you want and follow these instructions: viewtopic.php?t=7475&start=18
At the moment, the only way to see how much data you have for each symbol is to right click --> Edit Data and load the interval for which you have data.
I have forwarded your suggestions to the developers.
- arnie
- Posts: 1594
- Joined: 11 Feb 2009
- Location: Portugal
- Has thanked: 481 times
- Been thanked: 514 times
Re: Reaching the 4GB db limit.
Hi jek.
4gb of data is a lot of data.
Can you tell me what are we talking about (how many years of data, minute and daily, and how many symbols), just so I can have an idea.
Regards.
4gb of data is a lot of data.
Can you tell me what are we talking about (how many years of data, minute and daily, and how many symbols), just so I can have an idea.
Regards.
Re: Reaching the 4GB db limit.
So I hear that this process results in creating a completely new database that doesn't have any of your stored data. Is this true?To reduce your database you need to delete symbols that you want and follow these instructions: viewtopic.php?t=7475&start=18
This is a huge problem since the data providers often don't have tick data back more than a couple of weeks.
We would have to manually go through every contract and export it and then go back and import it into the new database if we want to use it. True?
That is a huge amount of work.
You should have a simple COM API to export and import symbols!
Or a way to run QuoteManager to with arguments list/import/export symbols so that it can be scripted.
viewtopic.php?f=1&t=4471
viewtopic.php?f=1&t=6761
Re: Reaching the 4GB db limit.
MultiCharts should store the actual tick data in the filesystem and just use Firebird for all the metadata. This would completely eliminate the database 4GB limit.
Pros/Cons for converting from the database to filesystem map:
Pros:
Pros/Cons for converting from the database to filesystem map:
Pros:
- Immediately break the 4GB limit for the database and replace it with a limit of 4GB per symbol/resolution. With 64-bit file pointers, even this wouldn't be a limit.
Eliminate a lot of complexity as well as dependence on external technology that may develop issues with platforms of interest to TSS.
Won't have to wait for tsserver.exe to exit before shutting down to copy saved data into database.
Backups of databases would only need to update those data files that changed. And using rsync for backups, only the ends of files would have to be transmitted.
Can share data collected from different machines. Specifically, you can have one computer collect futures data and another collect stock data and it is possible at the end of the day to update each of them to have a consistent database. I'm constantly having to export symbols and send move them around. If I could just copy the file, it would save me a *lot* of time.
Deleting, listing, or copying symbols would be incredibly simple and obvious. In fact, much of the pain of dealing with market data would be reduced.
- Searching for symbols becomes a filesystem walk which can be slower. But we aren't talking millions of symbols.
The database is "transactional" and durable. Note, however, that filesystems also have the ability to version files and can rollback to previous versions on a failure.
Already built whereas converting to filesystem requires work.
- Dave Masalov
- Posts: 1712
- Joined: 16 Apr 2010
- Has thanked: 51 times
- Been thanked: 489 times
Re: Reaching the 4GB db limit.
Dear jek,So I hear that this process results in creating a completely new database that doesn't have any of your stored data. Is this true?
This is a huge problem since the data providers often don't have tick data back more than a couple of weeks.
We would have to manually go through every contract and export it and then go back and import it into the new database if we want to use it. True?
Your data will not be deleted. The size of the database will be reduced at the expense of the data that you have already deleted. You will not have to manually export or import anything.
Your suggestion has been forwarded to the developers.MultiCharts should store the actual tick data in the filesystem and just use Firebird for all the metadata. This would completely eliminate the database 4GB limit.
- JoshM
- Posts: 2195
- Joined: 20 May 2011
- Location: The Netherlands
- Has thanked: 1544 times
- Been thanked: 1565 times
- Contact:
Re: Reaching the 4GB db limit.
I have some dummy questions about the database which I hope someone can answer.
- Is the 4GB limit still valid for MC 8.x?
- Is the limit for MC 64-bit also 4GB?
- Can a MC 32-bit database be imported/used in MC 64-bit?
- Is there a 'performance penalty' for a big database, or wouldn't this matter to MC?
- Is the database of MC .NET safely interchangeable (with manual importing) with a database of regular MC?
- Is the 4GB limit still valid for MC 8.x?
- Is the limit for MC 64-bit also 4GB?
- Can a MC 32-bit database be imported/used in MC 64-bit?
- Is there a 'performance penalty' for a big database, or wouldn't this matter to MC?
- Is the database of MC .NET safely interchangeable (with manual importing) with a database of regular MC?
- TJ
- Posts: 7743
- Joined: 29 Aug 2006
- Location: Global Citizen
- Has thanked: 1033 times
- Been thanked: 2223 times
Re: Reaching the 4GB db limit.
How to Migrate from MultiCharts 32 bit to MultiCharts 64 bitI have some dummy questions about the database which I hope someone can answer.
...
- Can a MC 32-bit database be imported/used in MC 64-bit?
https://www.multicharts.com/trading-sof ... rts_64_bit
- TJ
- Posts: 7743
- Joined: 29 Aug 2006
- Location: Global Citizen
- Has thanked: 1033 times
- Been thanked: 2223 times
Re: Reaching the 4GB db limit.
I don't believe there is a performance penalty for a big database, because MultiCharts works off cache in real time.I have some dummy questions about the database which I hope someone can answer.
...
- Is there a 'performance penalty' for a big database, or wouldn't this matter to MC?
...
MultiCharts saves the cache to your database when you shut down MultiCharts.
Re: Reaching the 4GB db limit.
I reduced my database from 2.8GB to 1.2GB by exporting all symbols and data, deleting the original databases (after backup), setting up new db's and importing all data again. I figured out this way, because never saw the storage database reducing the size after deleting symbols and data.
So I tried this different way. That is all ....
Greetings Mike
So I tried this different way. That is all ....
Greetings Mike
- Dave Masalov
- Posts: 1712
- Joined: 16 Apr 2010
- Has thanked: 51 times
- Been thanked: 489 times
Re: Reaching the 4GB db limit.
Mike,I reduced my database from 2.8GB to 1.2GB by exporting all symbols and data, deleting the original databases (after backup), setting up new db's and importing all data again. I figured out this way, because never saw the storage database reducing the size after deleting symbols and data.
So I tried this different way. That is all ....
Greetings Mike
Please see the following article on how to reduce the database: https://www.multicharts.com/trading-sof ... abase_Size