Array Size - Maximum

Questions about MultiCharts and user contributed studies.
Meecc
Posts: 50
Joined: 23 Jun 2011
Has thanked: 22 times
Been thanked: 7 times

Array Size - Maximum

Postby Meecc » 09 Aug 2014

Is there max limit size for 1D dynamic array that can be dynamically set ?

Is there max limit size for 1D 2D 3D normal array ?

Do we have 4D 5D array ?
if possible, what are the maximum dimension we can do ?

Assume using a i7gen4 PC with ram for the cases of 8 GB and 16 GB,
practically, at above what level that the array size become too large and start to deteriorate MC calculation performance , and that we should avoid going beyond ?

For purpose of increasing the amount of boxes to keep data,
which has more impact on calculation performance, increase the array size alone or increase the amount dimension ?

By the way,
can we install MC 64 2 versions on the same machine eg. having both release version to trade and beta for testing ?

Thank you ...

User avatar
JoshM
Posts: 2195
Joined: 20 May 2011
Location: The Netherlands
Has thanked: 1544 times
Been thanked: 1565 times
Contact:

Re: Array Size - Maximum

Postby JoshM » 10 Aug 2014

Is there max limit size for 1D dynamic array that can be dynamically set ?
The number of elements in an array can be fixed or dynamic (unlimited).
(I doubt it really is unlimited, but that's what the wiki says).
Do we have 4D 5D array ?
if possible, what are the maximum dimension we can do ?
The maximum number of array dimensions in PowerLanguage is 9.
Source: Arrays - Wiki.

Meecc
Posts: 50
Joined: 23 Jun 2011
Has thanked: 22 times
Been thanked: 7 times

Re: Array Size - Maximum

Postby Meecc » 10 Aug 2014

Thank JoshM for your answers.

I have tried your code on
http://www.multicharts.com/discussion/v ... 06#p107290
calculate array size time by 50 times in for loops.
My PC i7 3.4GHz boostable to 3.9GHz
the results are

For array size of 10000
Fixed size array took 4.2170 seconds.
Dynamic array took 8.3060 seconds.

For array size of 20000
Fixed size array took 16.7960 seconds.
Dynamic array took 33.3140 seconds.

For array size of 30000
Fixed size array took 37.3160 seconds.
Dynamic array took 74.1780 seconds.

For array size of 40000
Fixed size array took 65.2080 seconds.
Dynamic array took 130.1290 seconds.

For array size of 50000
Fixed size array took 101.9010 seconds.
Dynamic array took 204.7170 seconds.

then I change to size of 1 million
Var: arrayCount(1000000)
Arrays: fixedSize[1000000](0)
It took more than 30 mins and still calculating , so I close the workspace.

In general for one bar calculation to use 3 mins to calculate, it is way too long.

In case we need to calculate in all bars,
it is better to keep in maximum of less than 15 seconds a bar to be tolerable, in my sense.

Thank you for your codes Josh .


In using array with dimension increased from 1D to higher dimension,
there is not many functions and apparently inadequate functions to work on 2D array,
not to mention any higher than 2D dimensions which there is none at all.

Does anyone backtest or optimize eg. 2 yrs data of tick level bars , wait for 10 -24 hrs and still MC does not hang and return correct value ?
When wait for that long it not resistable to assume it is hanged already
and stop the calculation.
What can we put in to the test to still feel confident that it is not hang already ?

Is there a generally good recommendation to work on bigger calculated data ?
The calculated data is kept in arrays and is not the basic OHLCV of the bar.

User avatar
TJ
Posts: 7740
Joined: 29 Aug 2006
Location: Global Citizen
Has thanked: 1033 times
Been thanked: 2221 times

Re: Array Size - Maximum

Postby TJ » 10 Aug 2014

::
By the way,
can we install MC 64 2 versions on the same machine eg. having both release version to trade and beta for testing ?

Thank you ...
Each MultiCharts license allows you to run one instance of the software at any one time.

You can install MultiCharts in unlimited number of machines, but you can only run one at a time.

You can co-install in the same machine various MultiCharts flavors. ie 64 bit version, 32 bit verson, and .net version, but only one release of each version. ie you cannot install MC64 ver 8 and ver 9 in the same machine at the same time. Ver 9 will overwrite ver 8 during installation.

Meecc
Posts: 50
Joined: 23 Jun 2011
Has thanked: 22 times
Been thanked: 7 times

Re: Array Size - Maximum

Postby Meecc » 10 Aug 2014

You can co-install in the same machine various MultiCharts flavors. ie 64 bit version, 32 bit verson, and .net version,

but only one release of each version. ie you cannot install MC64 ver 8 and ver 9 in the same machine at the same time.
Thank you TJ ,

it is now clear that in one machine only one release for each version.

User avatar
TJ
Posts: 7740
Joined: 29 Aug 2006
Location: Global Citizen
Has thanked: 1033 times
Been thanked: 2221 times

Re: Array Size - Maximum

Postby TJ » 10 Aug 2014

::
then I change to size of 1 million
Var: arrayCount(1000000)
Arrays: fixedSize[1000000](0)
It took more than 30 mins and still calculating , so I close the workspace.

In general for one bar calculation to use 3 mins to calculate, it is way too long.
::.
1 million elements !?!?

I know that's only a test... but,
Array was never designed to be used as a "database".
Have you ever loaded a 1 million bars chart? On any charting software? How was the performance?
How about a retrieve and recalc of 1 million cells on an Excel worksheet? How long did it take?

It is ok to have high expectation on your software,
but everything must be put into perspective and context.

An array is a memory resident temporary "store".
The limitation is more on the memory you have (speed and size), and the efficiency of your motherboard. If you have a gaming machine, you will probably see slightly better result than a store bought student computer.


Having said the above, I regularly run an indicator with a 400x10 array, with no noticeable latency.
YMMV

Meecc
Posts: 50
Joined: 23 Jun 2011
Has thanked: 22 times
Been thanked: 7 times

Re: Array Size - Maximum

Postby Meecc » 11 Aug 2014

Thank you for your info, TJ.
I just would like to know the upper limit of MC.
1 million elements !?!?
I know that's only a test... but, Array was never designed to be used as a "database".

Have you ever loaded a 1 million bars chart? On any charting software? How was the performance?

I Just tested 2.5 years 2-point bar that built up to over 2 million bars.

MC process alone took 2.4 GB RAM , from running simple backtest
to show the Performance report with Drawdown chart.
It took 3 min which is longer than usual, just to show equity curve with DD chart.
With this, there is no array involved yet at all.

Here MC did not hang ,
although shown as paled window and not responding in task manager for 3 minute,
which almost made me to believe as already hang, still finally it finished the calculation.

Still looking for hearing recommendation and experiences
for best practice database handling in MC.

Array seem better choice than
ADE which do not support multicore optimization in MC.

User avatar
JoshM
Posts: 2195
Joined: 20 May 2011
Location: The Netherlands
Has thanked: 1544 times
Been thanked: 1565 times
Contact:

Re: Array Size - Maximum

Postby JoshM » 30 Aug 2014

::
then I change to size of 1 million
Var: arrayCount(1000000)
Arrays: fixedSize[1000000](0)
It took more than 30 mins and still calculating , so I close the workspace.

In general for one bar calculation to use 3 mins to calculate, it is way too long.
::.
1 million elements !?!?

I know that's only a test... but,
Array was never designed to be used as a "database".
Have you ever loaded a 1 million bars chart? On any charting software? How was the performance?
How about a retrieve and recalc of 1 million cells on an Excel worksheet? How long did it take?

It is ok to have high expectation on your software,
but everything must be put into perspective and context.
1 million array integer elements for a one-dimensional C# array for example only takes 4mb of memory. That cannot really be compared to the overhead added by 1 million bar charts or 1 million cells in Excel.

Just saying that the array size is not causing the high memory usage here, but the MC program is when using 1-2 million price bars.

Meecc
Posts: 50
Joined: 23 Jun 2011
Has thanked: 22 times
Been thanked: 7 times

Re: Array Size - Maximum

Postby Meecc » 30 Aug 2014

::
then I change to size of 1 million
Var: arrayCount(1000000)
Arrays: fixedSize[1000000](0)
It took more than 30 mins and still calculating , so I close the workspace.

In general for one bar calculation to use 3 mins to calculate, it is way too long.
::.
1 million elements !?!?

I know that's only a test... but,
Array was never designed to be used as a "database".
Have you ever loaded a 1 million bars chart? On any charting software? How was the performance?
How about a retrieve and recalc of 1 million cells on an Excel worksheet? How long did it take?

It is ok to have high expectation on your software,
but everything must be put into perspective and context.
1 million array integer elements for a one-dimensional C# array for example only takes 4mb of memory. That cannot really be compared to the overhead added by 1 million bar charts or 1 million cells in Excel.

Just saying that the array size is not causing the high memory usage here, but the MC program is when using 1-2 million price bars.
Just to point that I tested the above result using JoshM's script
that calculated on only the last bar
with for loops ran 1 million rounds that made MC hanged.
My 16 GB Ram was ample there and was not the problem.

For further clarification:

As to above JoshM's comment, it will use only 4 mb ram for 1 million array size.

If we declared array to 1 million in size and set default value to 0,
but actually use only 1000 array items,
this is to compare the affect on ram usage when
declare huge - use little and
declare little - use little.

That is, if is not the issue at all to declare huge and use little ,
so we can set huge array size at declaration and left unchanged at all time
for any further increase in array size usage.

Does it take ram space for all the 1 million array items at declaration,
or it takes ram gradually as it actually called, increased in the usage size in the running script ?
This is for static array.

Update Sep 02 :
Just learned that, by declaration of array size ,
contiguous RAM block is allocated for the array declared.
Last edited by Meecc on 01 Sep 2014, edited 1 time in total.


Return to “MultiCharts”