CPU Usage During Optimization

Questions about MultiCharts and user contributed studies.
User avatar
fbertram
Posts: 166
Joined: 16 Oct 2014
Location: Seattle, USA
Has thanked: 36 times
Been thanked: 73 times
Contact:

CPU Usage During Optimization

Postby fbertram » 22 May 2015

Dear MultiCharts fans,

I am in the process of optimizing a strategy with Portfolio Trader. This is a big task, so every little bit of time savings I can make will help. I noticed a few things:

* initial loading of the data takes quite some time (I am simulating ~200 stocks, 5 minute charts, 10 years). Unfortunately, it seems that MultiCharts is using only a single core to load all these data, instead of using the 16 cores.

* when doing exhaustive optimization, all cores are nicely utilized.

* when doing genetic optimization, this doesn't seem to be the case:
- with the 5m charts mentioned above, I see 15 of my 16 cores fully utilized; Portfolio Trader seems to run 15 simulations in parallel. The last core may or may not be used for general housekeeping, at least I didn't see it ever being used for more than 50%.
- with daily charts, the simulations are about 1 minute each and I see a completely different pattern: None of the cores are utilized to 100% and overall CPU load is never more than 45%. This is actually not a good use of a fast machine. See the picture below.

* when doing backtesting, Portfolio Trader really doesn't seem to care for multiple cores. This is sad, as the individual strategies could be calculated in parallel, before the MM signal combines the results.

Now to my questions:
* is there anything that I can do to increase usage of the cores, especially during data loading and optimization?
* are there any improvements planned for future MultiCharts versions?


Thank you,
best regards, Felix

Image
Attachments
Screen Shot 2015-05-22 at 10.30.33 AM.png
Screen Shot 2015-05-22 at 10.30.33 AM.png (185.73 KiB) Viewed 14021 times

User avatar
JoshM
Posts: 2099
Joined: 20 May 2011
Location: The Netherlands
Has thanked: 1526 times
Been thanked: 1502 times
Contact:

Re: CPU Usage During Optimization

Postby JoshM » 22 May 2015

* are there any improvements planned for future MultiCharts versions?
I can only guess from the outside looking in, but the feature request to have higher optimisation and backtesting speed by implementing new technology such as GPU calculation has been declined earlier this week:

MC-1735 - Hope MC could get much higher optimization/backtesting speed by using new technology such as GPU calculation.

User avatar
fbertram
Posts: 166
Joined: 16 Oct 2014
Location: Seattle, USA
Has thanked: 36 times
Been thanked: 73 times
Contact:

Re: CPU Usage During Optimization

Postby fbertram » 22 May 2015

well, I am not necessarily looking for GPU optimizations, these probably come with their very own set of challenges. All I am asking for is to use multi-cores more efficiently. I very well understand that for single strategies, that is not easy. For Portfolio Trader as well as optimization the situation is different though: there are a lot of things that *could* be done in parallel- but unfortunately aren't. So having a 16-core Xeon available actually doesn't help that much...

Cheers, Felix

User avatar
Henry MultiСharts
Posts: 9121
Joined: 25 Aug 2011
Has thanked: 1259 times
Been thanked: 2927 times

Re: CPU Usage During Optimization

Postby Henry MultiСharts » 26 May 2015

Hello Felix,

The whole MultiCharts application uses sequential calculations (charting, optimization, backtesting).
Video card does not affect MultiCharts performance as GPU cannot be utilized for sequential calculations (that is not possible to divide the data series into parts and do the calculations without referencing historical data).

The speed of strategies calculation/optimization is increased when using a multi-core CPU, as the calculations are divided into multiple flows. Multi-core CPU will also increase chart indicator calculation as the calculations are split between the cores (all studies from a single chart are calculated in a separate thread).
Multi-core CPU will not increase the backtesting speed as it is a linear calculation procedure and it cannot be divided into parts for calculation. Here is a good description from TJ.

If you want to force MultiCharts to utilize 16 cores for optimization you can import the attached reg key. Unzip the file, run it, confirm registry modification. The changes will come into effect after all MultiCharts processes are restarted.
Attachments
16_Threads_Optimization.zip
(276 Bytes) Downloaded 385 times

signalworks
Posts: 54
Joined: 06 Oct 2013
Location: Germany.Solingen
Has thanked: 20 times
Been thanked: 2 times
Contact:

Re: CPU Usage During Optimization

Postby signalworks » 12 Oct 2015

Dear MC-Team,

I use the MultiCharts64 Version 9.0 Release (Build 11210) on Win7/64Bit and have problems with using all cores while backtesting. I have installed your Registry-hack "All_Thread_ForOptimize.reg". This works fine, if MC is new started. But after some tests (I could not determine a definitely relation yet), only 5-6 thread are further used. When I close MC and re-install the registry-hack, it works fine again. Is there something, that overwrites the hack from the system OS perhaps?

Thanks.

regards,
Lutz
Attachments
cpu_03.PNG
cpu_03.PNG (14.53 KiB) Viewed 13564 times

User avatar
TJ
Posts: 7213
Joined: 29 Aug 2006
Location: Global Citizen
Has thanked: 991 times
Been thanked: 2052 times

Re: CPU Usage During Optimization

Postby TJ » 12 Oct 2015

Dear MC-Team,

I use the MultiCharts64 Version 9.0 Release (Build 11210) on Win7/64Bit and have problems with using all cores while backtesting. I have installed your Registry-hack "All_Thread_ForOptimize.reg". This works fine, if MC is new started. But after some tests (I could not determine a definitely relation yet), only 5-6 thread are further used. When I close MC and re-install the registry-hack, it works fine again. Is there something, that overwrites the hack from the system OS perhaps?

Thanks.

regards,
Lutz
Are you backtesting or optimizing? You seemed to say both.

[FAQ] Backtesting vs Optimization CPU Usage -- Single Core vs Multi-Core
See post #13
viewtopic.php?f=16&t=10811

signalworks
Posts: 54
Joined: 06 Oct 2013
Location: Germany.Solingen
Has thanked: 20 times
Been thanked: 2 times
Contact:

Re: CPU Usage During Optimization

Postby signalworks » 12 Oct 2015

sorry, uncleared expressed, should called above: "while optimization".
Thanks.

User avatar
Henry MultiСharts
Posts: 9121
Joined: 25 Aug 2011
Has thanked: 1259 times
Been thanked: 2927 times

Re: CPU Usage During Optimization

Postby Henry MultiСharts » 15 Oct 2015

Dear MC-Team,

I use the MultiCharts64 Version 9.0 Release (Build 11210) on Win7/64Bit and have problems with using all cores while backtesting. I have installed your Registry-hack "All_Thread_ForOptimize.reg". This works fine, if MC is new started. But after some tests (I could not determine a definitely relation yet), only 5-6 thread are further used. When I close MC and re-install the registry-hack, it works fine again. Is there something, that overwrites the hack from the system OS perhaps?

Thanks.

regards,
Lutz
Hello Lutz,

Are you optimizing the same strategy on the same chart in all these tests?

signalworks
Posts: 54
Joined: 06 Oct 2013
Location: Germany.Solingen
Has thanked: 20 times
Been thanked: 2 times
Contact:

Re: CPU Usage During Optimization

Postby signalworks » 15 Oct 2015

Hello Henry,
yes, same strategy in this case. But that is independent, with other strategies it is the same effect.
Maybe there is a hint with changing chartsettings or create a new one/new workspace. I will try something with it.

regards,
Lutz

User avatar
Henry MultiСharts
Posts: 9121
Joined: 25 Aug 2011
Has thanked: 1259 times
Been thanked: 2927 times

Re: CPU Usage During Optimization

Postby Henry MultiСharts » 19 Oct 2015

Hello Henry,
yes, same strategy in this case. But that is independent, with other strategies it is the same effect.
Maybe there is a hint with changing chartsettings or create a new one/new workspace. I will try something with it.

regards,
Lutz
Hello Lutz,

Please send us (support@multicharts.com) the following information for analysis:
- workspace you are using;
- in QuoteManager select the symbols you are using, make a right click on the selection->Export data->Export instrument (with data). Send us the QMD export file for analysis;
- in PowerLanguage editor->File->Export->export the studies you are using for replicating this behavior; send us the export file;
- specify the version and build number of MultiCharts you are running (in MultiCharts go to Help tab-> About);
- instructions for replicating this behavior;
- screenshot of your system specs (Control Panel\System\General or right click on the Computer icon and select properties)
- screenshots or video demonstrating this case.

If the file size is >10 mb please upload it to any file sharing hosting and send us the download link.


Return to “MultiCharts”