Genetic Optimization settings and behaviour in relation to number of cores  [SOLVED]

Questions about MultiCharts and user contributed studies.
wilkinsw
Posts: 378
Joined: 21 Apr 2013
Has thanked: 92 times
Been thanked: 59 times

Genetic Optimization settings and behaviour in relation to number of cores

Postby wilkinsw » 05 Feb 2018

Hi,

I uncovered some interesting behaviour that I had never noticed before when running a genetic optimization with varying number of cores.

I would really appreciate some guidance on this as I want to make sure that increasing cores doesn't mean reducing GA efficacy.

Firstly a link to all one needs to reproduce:

https://drive.google.com/drive/folders/ ... sp=sharing

The problem:

Default "Maximum number of generations" reduces in direct proportion with number of cores on the machine. This then maintains number of iterations needed to complete the optimization (# of simulations : 11314(11318 on 4 core)). In other words, "Maximum number of generations" should actually be called "Maximum number of generations PER CORE".

sims.PNG
sims.PNG (6.29 KiB) Viewed 778 times


Total simulations (11314) = generations x cores + starting population !

I had no idea that each step of evolution (new generation) is being delegated to a core like this.

Surely we are just creating unrelated, parallel genetic optimizations? So the more cores we have, the more parallel, unrelated genetic optimizations taking place? So, in the above example, if I used a massive 10560 cores then MC's genetic optimizer will 1) run 754 parallel backtests (= generation 1) then 2) run another 10560 backtests (=generation 2), each one on a seperate core 3) END of optimization (with virtually no evolution whatsoever: an unfit solution).

If I've got that right, can we see the problem there? Genetic algorithm efficacy (and relevancy) diminishes in direct proportion to cores used! Sure, it's a faster optimization with more cores, but we've lost quality and reduced the number of sequential steps of signal evolution.

Is this expected behaviour? Is there another setting I should be using to have each generation created sequentially, directly descended from the prior generation, with all cores being used to work on the current generation only (not parallel optimisations)?

Thanks!

4 core default settings (11318 iteration optimization):
4 core GA default settings  2641 generations.PNG
4 core GA default settings 2641 generations.PNG (18.39 KiB) Viewed 783 times


8 core default settings (11314 iteration optimization):
8 core GA default settings  1320 generations.PNG
8 core GA default settings 1320 generations.PNG (14.46 KiB) Viewed 783 times


40 core default settings (11314 iteration optimization):
40 core GA default settings  264 generations.PNG
40 core GA default settings 264 generations.PNG (15.84 KiB) Viewed 783 times

wilkinsw
Posts: 378
Joined: 21 Apr 2013
Has thanked: 92 times
Been thanked: 59 times

Re: Genetic Optimization settings and behaviour in relation to number of cores

Postby wilkinsw » 05 Feb 2018

I've run some more tests and the results have confirmed my worst fears:

The more cores used, the poorer the quality of genetic optimization.

In MC, it would certainly appear that each core gets given its own genetic optimization. Therefore the more cores you use the more parallel genetic optimizations run. This merely results in a larger number of less fit solutions. So the more cores you use, the more you move towards a Brute Force optimization and the more you lose the benefits of the GA! I might be wrong, but the results speak for themselves.

Evidence:

I ran 2 parallel studies:

a) 10 core machine: 10 optimizations on a large space (500 million permutation space) using an incremental GA that was only allowed to run 250 simulations.

b) 40 core machine: 10 optimizations on a large space (500 million permutation space) using an incremental GA that was only allowed to run 250 simulations.

In MC: number of simulations = first generation population + generations * cores.

Therefore arm A had the settings of population 50 and generations 50 and B had the settings of population 50 and generations 5.

I then collected the average, min and max fitness results (net profit) for each test (10 tests for A and 10 tests for B).

Results:
results.PNG
results.PNG (3.33 KiB) Viewed 766 times


You can see that the 4 core machine was vastly superior vs 40 cores at finding the fittest population. The best result for B (40 cores) was much less than the worst for A! The 40 cores machine was obviously much much faster though.

Conclusions:

The incremental GA in MC is very powerful. But, currently, it is not optimized to be used on multiple cores, unless the user is happy with diminished benefits of GA vs fewer cores. The only way around this is to simply turn up the number of generations, but this comes at the cost of time. The user would therefore be better off using a cheaper machine with fewer cores. Many users will be looking at machines with more and more cores as costs decrease. Unless using brute force optimizations, there's currently no benefit in doing so.

Suggestions:

"Number of generations" should be independant of cores. It should be related to the algorithm only and not be for setting up parallel processes. The algorithm should then make best use of all cores whilst maximising the benefits of the GA.

If, and only if, my assumptions are correct: Multicharts need to rework their incremental GA.

In the incremental GA, from generation 2 onwards, each simulation is simply a backtest of the new child to be added to the population (replacing an exisiting member "genome"). The child is generated by genome mutation and/or crossover (in accordance with the user defined settings). Currently this process occurs in parallel (I believe). So, 40 cores = 40 independent processes, not receiving the same "learning" effect of a sequential process.

It is vital that children are generated and existing population genomes replaced sequentially. Is there a way of doing this? Maybe instead of 1 child, maybe number of children could equal cores? The best (fittest) child gets kept and replaces an original? Maybe all children that are superior to the user defined "parent/worst/random" get kept?

https://stackoverflow.com/questions/317 ... core-speed

User avatar
TJ
Posts: 6839
Joined: 29 Aug 2006
Location: Global Citizen
Has thanked: 979 times
Been thanked: 1968 times

Re: Genetic Optimization settings and behaviour in relation to number of cores

Postby TJ » 05 Feb 2018

Are you talking about real cores? or virtual cores?

wilkinsw
Posts: 378
Joined: 21 Apr 2013
Has thanked: 92 times
Been thanked: 59 times

Re: Genetic Optimization settings and behaviour in relation to number of cores

Postby wilkinsw » 05 Feb 2018

very real

User avatar
Anna MultiCharts
Posts: 206
Joined: 14 Jul 2017
Has thanked: 10 times
Been thanked: 53 times

Re: Genetic Optimization settings and behaviour in relation to number of cores

Postby Anna MultiCharts » 08 Feb 2018

Hello, wilkinsw!

Our engineers are analyzing this case on our side. I'll let you know the results as soon as they are available.

wilkinsw
Posts: 378
Joined: 21 Apr 2013
Has thanked: 92 times
Been thanked: 59 times

Re: Genetic Optimization settings and behaviour in relation to number of cores

Postby wilkinsw » 08 Feb 2018

Thanks Anna.

Will be great to know the exact process for how an incremental GO is run on multi cores in MC.

I'm also running further tests regarding incremental vs basic vs basic elitist on multi cores my side and will try and report back next week.

GuppyDRV
Posts: 30
Joined: 20 Jan 2017
Been thanked: 1 time

Re: Genetic Optimization settings and behaviour in relation to number of cores

Postby GuppyDRV » 11 Feb 2018

Wilkinsw,

I arrived at a similar conclusion without completely understanding the why. My solution was much the same as yours, increase the number of generations. At 3x generations I have found consistent results. At 10x absolutely consistent results. I also tested with very high iteration numbers, far beyond anything I would ever use., just to prove reliability. The sacrifice is speed. I will be interested to hear more regarding your findings using different GO optimization settings as well as direct MC developer feedback.

Thanks for taking the time to post this it's really important information for those using GO and high core count systems.

GuppyDRV
These users thanked the author GuppyDRV for the post:
wilkinsw

wilkinsw
Posts: 378
Joined: 21 Apr 2013
Has thanked: 92 times
Been thanked: 59 times

Re: Genetic Optimization settings and behaviour in relation to number of cores

Postby wilkinsw » 13 Feb 2018

I've been running lots of tests: MC and PT based optimizations and WFAs.

I haven't found anything slam dunk conclusive but, anecdotally would report the following:

-Incremental GA is consistently better than basic GA.
-With incremental GA; fewer cores and more generations (same overall number of simulations) on balance seems better than more cores and less generations (same overall number of simulations).
-I found basic GA might be of use when one wants to find a solution as quickly as possible, but sacrifices quality: when I ran WFAs but with very few simulations permitted (due to time constraints), on many metrics (OOS performance especially) GA basic at least matched GA incremental. Therefore I suspect GA basic is good for "fishing" in very large optimization spaces (in relation to the number of simulations permitted).

Can MC or anyone else recommend any 3rd party addons that might provide a more elaborate Genetic optimizer?

User avatar
Anna MultiCharts
Posts: 206
Joined: 14 Jul 2017
Has thanked: 10 times
Been thanked: 53 times

Re: Genetic Optimization settings and behaviour in relation to number of cores

Postby Anna MultiCharts » 14 Feb 2018

Wilkinsw,

Our engineers have tested this case on our side and provided a reply.

1) “Maximum number of generations” should indeed be perceived as “Maximum number of generations per core”.
Actually genetic optimization has one hidden parameter in the Registry:
HKEY_CURRENT_USER\Software\TS Support\Power Language\StudyRunner -> NumberOfThreadsOnOptimization (DWORD)

It is the value by which the “Maximum number of generations” value is multiplied.
If this key is missing in the registry (which is by default) then the multiplier equals the number of logical cores.
When there’s no key, then the parameters for the fast optimization are set, so on different machines with different core number the settings will also be different.

2) If you compare Genetic optimization on different machines then you need to set identical optimization parameters taking into account the amount of cores OR change the value in the registry key. Your tests show that the optimization was carried out with different settings, even though you entered the same values in the optimization properties dialogue: the amount of cores implicitly influenced the optimization.

3) “In MC, it would certainly appear that each core gets given its own genetic optimization. Therefore the more cores you use the more parallel genetic optimizations run. This merely results in a larger number of less fit solutions. So the more cores you use, the more you move towards a Brute Force optimization and the more you lose the benefits of the GA! I might be wrong, but the results speak for themselves” – next generation always waits until all individuals of the previous generation are calculated. Two generations cannot be calculated simultaneously.

4) In the next MultiCharts version we’ll change this behavior:
- the “Maximum number of generations” value will be multiplied by the number of cores and will be displayed this way;
- it will be possible to change the amount of cores accessible for optimization from the Optimization progress dialogue so that users could assign PC’s resources for other tasks at their will while the optimization is still in progress.

wilkinsw
Posts: 378
Joined: 21 Apr 2013
Has thanked: 92 times
Been thanked: 59 times

Re: Genetic Optimization settings and behaviour in relation to number of cores

Postby wilkinsw » 14 Feb 2018

Hi Anna!

next generation always waits until all individuals of the previous generation are calculated. Two generations cannot be calculated simultaneously.


It is my understanding that in incremental GA the next generation is merely the previous generation plus with one member changed only. So your point about waiting for "all individuals" doesn't make sense to me. Ie the next generation is one backtest only.

Therefore what you have stated confirms my findings: because the incremental GA has to wait for the result of each and every single backtest (unlike the basic GA, which required multiple backtest to create a new population each generation), it makes the utilization of multiple cores a challenge.


One of my questions remains:

MC is clearly doing something in parallel when running an incremental GA (it is not running backtests/generations sequentially) on multiple cores. What processes are occurring in parallel?

Ie can you or someone at MC map out the incremental GA's progression from the creation of generation 1 to generation 2 onwards? How is that process delegated across multiple cores?

Thanks.

wilkinsw
Posts: 378
Joined: 21 Apr 2013
Has thanked: 92 times
Been thanked: 59 times

Re: Genetic Optimization settings and behaviour in relation to number of cores

Postby wilkinsw » 19 Feb 2018

Hi Anna,

In addition to my last post, can you advise on the following please:

I'm probably going to ditch my 40 core machine. Before I do, is there a way of telling MC to only optimise using x no. cores for a GA?

For example, could I open up 40 instances of MC and run 40 incremental GAs, each one using only one core each?

User avatar
Anna MultiCharts
Posts: 206
Joined: 14 Jul 2017
Has thanked: 10 times
Been thanked: 53 times

Re: Genetic Optimization settings and behaviour in relation to number of cores

Postby Anna MultiCharts » 22 Feb 2018

wilkinsw,

in incremental GA a new generation is created after concurrent crossover of as many pairs from the previous population as allowed by the number of available threads (cores). If the population size is significantly bigger than the number of threads, then the result won’t be much different from the classic incremental GA.

If you need to use classic incremental GA then you can adjust the following key in the registry:
HKEY_CURRENT_USER\Software\TS Support\Power Language\StudyRunner -> NumberOfThreadsOnOptimization=2
In this case the number of cores will be limited and the time spent on optimization will increase if the actual number of cores is bigger than 2.

We’ve checked your case by changing this registry key value to 2, 4, 40, 80 with default GA settings, and with all of them the same NP maximum is reached (2113.20).

Most processors use hyperthreading, this means that the number of logical cores is twice as big as the number of physical ones.
Logical cores don’t give any advantages in speed.
Please try to set the NumberOfThreadsOnOptimization=the number of real cores on your PC. Then there will be less additional crossovers in comparison to the classic incremental GA, but the speed will remain the same.

wilkinsw wrote:In addition to my last post, can you advise on the following please:

I'm probably going to ditch my 40 core machine. Before I do, is there a way of telling MC to only optimise using x no. cores for a GA?

For example, could I open up 40 instances of MC and run 40 incremental GAs, each one using only one core each?

You can set the NumberOfThreadsOnOptimization value to 1, then the time required for optimization will be doubled compared to when it’s set to 2. As you said incremental GA should perform at least 2 calculations to get the next generation. When one thread is involved these calculations will be performed subsequently, not in parallel.

We don’t see any purpose for limiting the number of cores for optimization as, again, if the population size is significantly bigger that the number of threads, then the result won’t be much different from the classic incremental GA.
These users thanked the author Anna MultiCharts for the post (total 2):
wilkinswkagein

wilkinsw
Posts: 378
Joined: 21 Apr 2013
Has thanked: 92 times
Been thanked: 59 times

Re: Genetic Optimization settings and behaviour in relation to number of cores

Postby wilkinsw » 22 Feb 2018

Thanks Anna!

Your detailed answer is exactly what I was after.

Understanding exactly how incremental GA might change with cores means I can go away and factor that in when researching.

For other readers, to conclude: if you want to maximise the true benefit of incremental GA then you want a fairly minimal number of cores:population size ratio. If this ratio = 1 (e.g. 10 cores and population of 10) then you are effectively running a basic GA (which I believe is less effective for most opimizations). So IMHO I advise increasing population size with cores or turning down allocated cores with smaller population sizes.

Thanks again.
Last edited by wilkinsw on 22 Feb 2018, edited 1 time in total.
These users thanked the author wilkinsw for the post:
kagein

wilkinsw
Posts: 378
Joined: 21 Apr 2013
Has thanked: 92 times
Been thanked: 59 times

Re: Genetic Optimization settings and behaviour in relation to number of cores

Postby wilkinsw » 22 Feb 2018

I have one more question....

Most processors use hyperthreading, this means that the number of logical cores is twice as big as the number of physical ones.
Logical cores don’t give any advantages in speed.
Please try to set the NumberOfThreadsOnOptimization=the number of real cores on your PC. Then there will be less additional crossovers in comparison to the classic incremental GA, but the speed will remain the same.


Regarding your above point and as an example, how many real cores does this chip have:

e3-1246v3?
https://www.cpubenchmark.net/cpu.php?cp ... 40+3.50GHz

cores.PNG
cores.PNG (16.26 KiB) Viewed 515 times


Thanks

User avatar
fbertram
Posts: 161
Joined: 16 Oct 2014
Location: Seattle, USA
Has thanked: 37 times
Been thanked: 66 times
Contact:

Re: Genetic Optimization settings and behaviour in relation to number of cores

Postby fbertram » 22 Feb 2018

it's 4 'real' cores. Thanks to hyper-threading, it looks like 8 processors to the OS.

User avatar
TJ
Posts: 6839
Joined: 29 Aug 2006
Location: Global Citizen
Has thanked: 979 times
Been thanked: 1968 times

Re: Genetic Optimization settings and behaviour in relation to number of cores

Postby TJ » 22 Feb 2018

wilkinsw wrote:I have one more question....

Most processors use hyperthreading, this means that the number of logical cores is twice as big as the number of physical ones.
Logical cores don’t give any advantages in speed.
Please try to set the NumberOfThreadsOnOptimization=the number of real cores on your PC. Then there will be less additional crossovers in comparison to the classic incremental GA, but the speed will remain the same.


Regarding your above point and as an example, how many real cores does this chip have:

e3-1246v3?
https://www.cpubenchmark.net/cpu.php?cp ... 40+3.50GHz

cores.PNG

Thanks



https://ark.intel.com/products/80916/In ... e-3_50-GHz

Performance
# of Cores = 4
# of Threads = 8

.
Attachments
cores.PNG
cores.PNG (16.73 KiB) Viewed 488 times

wilkinsw
Posts: 378
Joined: 21 Apr 2013
Has thanked: 92 times
Been thanked: 59 times

Re: Genetic Optimization settings and behaviour in relation to number of cores

Postby wilkinsw » 23 Feb 2018

I'll run some tests restricting GA to no. cores and report back with the comparative speed results.

wilkinsw
Posts: 378
Joined: 21 Apr 2013
Has thanked: 92 times
Been thanked: 59 times

Re: Genetic Optimization settings and behaviour in relation to number of cores  [SOLVED]

Postby wilkinsw » 23 Feb 2018

Test results:

Firstly, I suspect, like me on all my PCs, people won't find the registry path to alter number of cores and will need to add it. I've attached 2 registry files that Henry shared on another thread;

All_Thread_ForOptimize.zip
(523 Bytes) Downloaded 8 times


I ran the tests on a 4 core (8 logical) CPU: i7 - 4790.

For each test I alternated between 4 cores and 8 hyperthreaded cores.

Using the benchmark files I shared at the top of this thread, running brute force optimizations on GOOG symbol......

When running a single optim 4 cores was about 0.7x speed vs 8 hyperthreaded cores.

When running 4 simultaneous optims 4 cores matched 8 cores. So it seems only under higher loads does the 4 real core limitation come into effect. Otherwise stripping MC back to only using real cores and not the full hyperthreaded cores on offer, is the slower option for simple, single optims.

I repeated the test in PT with one signal applied to a basket of instruments (this is the most common optim I perform). I repeated it again as a WFA. In all cases hyperthreading was faster.

Conclusions:

If your preference is incremental GA, opt for machines with cores <= 10% of your minimum populations size that you use. Therefore, try and maximise clock speed whilst keeping total number of cores within this metric.

If you have a machine with excessive number of cores be aware that to preserve optimal "learning" effect of incremental GA and to protect against running more unstable basic GAs (which is effectively happening when number of cores >10% of population size) you will need to turn down the number of cores allocated to an optimization. Maximising capacity of one’s server with excessive cores can still take place only if running very demanding optimizations (ie multiple optims in tandem). Otherwise I advise ditching the big server and opting for multiple smaller, higher clocked servers.

GuppyDRV
Posts: 30
Joined: 20 Jan 2017
Been thanked: 1 time

Re: Genetic Optimization settings and behaviour in relation to number of cores

Postby GuppyDRV » 23 Feb 2018

So how would increasing the number of generations effect results. I have found that multiplying this number by number of real cores has produced stable and repeatable results with incremental genetic optimization. This is not optimum by any stretch of the imagination as it greatly increases process time but it seems to work.

@henrey maybe you guys can chime in and provide some clear guidelines on this subject as this thread has really not addressed in my opinion the matter fullly.

@wilkinsw great job and thanks for the work.

So.........these are the questions.

What settings will produce the most accurate results?
What actually is happening with cores and parallel incremental genetic optimization?
What if anything do I need to do or instal to achieve optimum performance?

Thanks for the help.

GuppyDRV

wilkinsw
Posts: 378
Joined: 21 Apr 2013
Has thanked: 92 times
Been thanked: 59 times

Re: Genetic Optimization settings and behaviour in relation to number of cores

Postby wilkinsw » 26 Feb 2018

Hi GuppyDRV,

In the above I was investigating how the the incremental algorithm actually changes with cores.

I believe the answers to all your questions have already been answered above.

The number of generations you require will depend on the size and complexity of the optimization space you are investigating.

Your explicit questions:

What settings will produce the most accurate results?
Ans: With regards to incremental GA....... large populations size and large number of generations (nb large core numbers vs population size will degrade the incremental algorithm).

What actually is happening with cores and parallel incremental genetic optimization?
Ans: Please read previous posts. Instead of 1 new child per generation (created from the crossover process), number of children equals cores. This is a solution that lies between incremental and basic GA in its logic. It means a sacrifice in search efficiency for speed. The reduction in search efficiency is a function of cores/population. One child per generation will on average be higher quality than the average child from a batch of new children (hope that makes sense).

What if anything do I need to do or instal to achieve optimum performance?
Ans: I think that's been addressed above too. Make sure population size is large enough, else, if wanting to preserve incremental GA logic, use the registy files to reduce allocated cores. Seperately make sure your computer is setup to run with prolonged higher CPU loads (look at you power settings).


Return to “MultiCharts”