Reaching the 4GB db limit.

Questions about MultiCharts and user contributed studies.
jek
Posts: 181
Joined: 24 Dec 2006
Has thanked: 1 time
Been thanked: 2 times

Reaching the 4GB db limit.

Postby jek » 16 Oct 2010

As I near the 4GB limit, I find that I don't see any fields in QuoteManager to easily find out what symbols are taking storage.

Can you say how to go about this?

Ideally, you would have a simple COM interface that I can use through Visual Basic to iterate through all the symbols, query their properties, and import or export them. I've wanted that for a *long* time... It is my absolutely biggest pain point of managing data.

Also, I made the suggestion long ago that there really isn't any pressing need to keep the compressed bar data in a database. You really ought to just map that into the filesystem and then it could grow as big as the filesystem (which is a lot these days).

Comments?

Anyone else grappling with the 4GB limit?

User avatar
Dave Masalov
Posts: 1712
Joined: 16 Apr 2010
Has thanked: 51 times
Been thanked: 489 times

Re: Reaching the 4GB db limit.

Postby Dave Masalov » 11 Nov 2010

Dear jek,

To reduce your database you need to delete symbols that you want and follow these instructions: viewtopic.php?t=7475&start=18

At the moment, the only way to see how much data you have for each symbol is to right click --> Edit Data and load the interval for which you have data.

I have forwarded your suggestions to the developers.

User avatar
arnie
Posts: 1594
Joined: 11 Feb 2009
Location: Portugal
Has thanked: 481 times
Been thanked: 514 times

Re: Reaching the 4GB db limit.

Postby arnie » 11 Nov 2010

Hi jek.

4gb of data is a lot of data.

Can you tell me what are we talking about (how many years of data, minute and daily, and how many symbols), just so I can have an idea.

Regards.

jek
Posts: 181
Joined: 24 Dec 2006
Has thanked: 1 time
Been thanked: 2 times

Re: Reaching the 4GB db limit.

Postby jek » 06 Dec 2010

To reduce your database you need to delete symbols that you want and follow these instructions: viewtopic.php?t=7475&start=18
So I hear that this process results in creating a completely new database that doesn't have any of your stored data. Is this true?

This is a huge problem since the data providers often don't have tick data back more than a couple of weeks.

We would have to manually go through every contract and export it and then go back and import it into the new database if we want to use it. True?

That is a huge amount of work.

You should have a simple COM API to export and import symbols!

Or a way to run QuoteManager to with arguments list/import/export symbols so that it can be scripted.

viewtopic.php?f=1&t=4471
viewtopic.php?f=1&t=6761

jek
Posts: 181
Joined: 24 Dec 2006
Has thanked: 1 time
Been thanked: 2 times

Re: Reaching the 4GB db limit.

Postby jek » 06 Dec 2010

MultiCharts should store the actual tick data in the filesystem and just use Firebird for all the metadata. This would completely eliminate the database 4GB limit.

Pros/Cons for converting from the database to filesystem map:

Pros:
  • Immediately break the 4GB limit for the database and replace it with a limit of 4GB per symbol/resolution. With 64-bit file pointers, even this wouldn't be a limit.

    Eliminate a lot of complexity as well as dependence on external technology that may develop issues with platforms of interest to TSS.

    Won't have to wait for tsserver.exe to exit before shutting down to copy saved data into database.

    Backups of databases would only need to update those data files that changed. And using rsync for backups, only the ends of files would have to be transmitted.

    Can share data collected from different machines. Specifically, you can have one computer collect futures data and another collect stock data and it is possible at the end of the day to update each of them to have a consistent database. I'm constantly having to export symbols and send move them around. If I could just copy the file, it would save me a *lot* of time.

    Deleting, listing, or copying symbols would be incredibly simple and obvious. In fact, much of the pain of dealing with market data would be reduced.
Cons:
  • Searching for symbols becomes a filesystem walk which can be slower. But we aren't talking millions of symbols.

    The database is "transactional" and durable. Note, however, that filesystems also have the ability to version files and can rollback to previous versions on a failure.

    Already built whereas converting to filesystem requires work.

User avatar
Dave Masalov
Posts: 1712
Joined: 16 Apr 2010
Has thanked: 51 times
Been thanked: 489 times

Re: Reaching the 4GB db limit.

Postby Dave Masalov » 07 Dec 2010

So I hear that this process results in creating a completely new database that doesn't have any of your stored data. Is this true?

This is a huge problem since the data providers often don't have tick data back more than a couple of weeks.

We would have to manually go through every contract and export it and then go back and import it into the new database if we want to use it. True?
Dear jek,

Your data will not be deleted. The size of the database will be reduced at the expense of the data that you have already deleted. You will not have to manually export or import anything.
MultiCharts should store the actual tick data in the filesystem and just use Firebird for all the metadata. This would completely eliminate the database 4GB limit.
Your suggestion has been forwarded to the developers.

User avatar
JoshM
Posts: 2195
Joined: 20 May 2011
Location: The Netherlands
Has thanked: 1544 times
Been thanked: 1565 times
Contact:

Re: Reaching the 4GB db limit.

Postby JoshM » 08 Dec 2012

I have some dummy questions about the database which I hope someone can answer. :)

- Is the 4GB limit still valid for MC 8.x?

- Is the limit for MC 64-bit also 4GB?

- Can a MC 32-bit database be imported/used in MC 64-bit?

- Is there a 'performance penalty' for a big database, or wouldn't this matter to MC?

- Is the database of MC .NET safely interchangeable (with manual importing) with a database of regular MC?

User avatar
TJ
Posts: 7740
Joined: 29 Aug 2006
Location: Global Citizen
Has thanked: 1033 times
Been thanked: 2221 times

Re: Reaching the 4GB db limit.

Postby TJ » 08 Dec 2012

I have some dummy questions about the database which I hope someone can answer. :)

...
- Can a MC 32-bit database be imported/used in MC 64-bit?
How to Migrate from MultiCharts 32 bit to MultiCharts 64 bit
https://www.multicharts.com/trading-sof ... rts_64_bit

User avatar
TJ
Posts: 7740
Joined: 29 Aug 2006
Location: Global Citizen
Has thanked: 1033 times
Been thanked: 2221 times

Re: Reaching the 4GB db limit.

Postby TJ » 08 Dec 2012

I have some dummy questions about the database which I hope someone can answer. :)
...
- Is there a 'performance penalty' for a big database, or wouldn't this matter to MC?
...
I don't believe there is a performance penalty for a big database, because MultiCharts works off cache in real time.

MultiCharts saves the cache to your database when you shut down MultiCharts.

User avatar
piranhaxp
Posts: 241
Joined: 18 Oct 2005
Has thanked: 4 times
Been thanked: 30 times

Re: Reaching the 4GB db limit.

Postby piranhaxp » 08 Dec 2012

I reduced my database from 2.8GB to 1.2GB by exporting all symbols and data, deleting the original databases (after backup), setting up new db's and importing all data again. I figured out this way, because never saw the storage database reducing the size after deleting symbols and data.

So I tried this different way. That is all ....

Greetings Mike

User avatar
Dave Masalov
Posts: 1712
Joined: 16 Apr 2010
Has thanked: 51 times
Been thanked: 489 times

Re: Reaching the 4GB db limit.

Postby Dave Masalov » 13 Dec 2012

I reduced my database from 2.8GB to 1.2GB by exporting all symbols and data, deleting the original databases (after backup), setting up new db's and importing all data again. I figured out this way, because never saw the storage database reducing the size after deleting symbols and data.

So I tried this different way. That is all ....

Greetings Mike
Mike,

Please see the following article on how to reduce the database: https://www.multicharts.com/trading-sof ... abase_Size


Return to “MultiCharts”