Best computer for Genetic Algo optimization  [SOLVED]

Questions about MultiCharts and user contributed studies.
KhaosTrader
Posts: 186
Joined: 10 May 2012
Has thanked: 14 times
Been thanked: 10 times

Best computer for Genetic Algo optimization

Postby KhaosTrader » 15 Sep 2012

What is the best computer type for Genetic Algo (GA) optimization?

Should I go with XEON processor with 128 gig memory? or just a standard hex core intel processor with 64 gig memory?

Also are you guys going to implement grid computing architecture in the future where I can have multiple computers crunch on a GA optimization?

Currently I bought 2 licenses of MC (and I have two MC.nets also) -- one license is for me to trade with , and the other one just crunches optimizations throughout the day. I really would like to make my optimizations work faster so that I can test a wider parameter space.

Also I really would like it if i could save the ranges of optimizations, rather than have to enter them every time i restart mc, they should be saved in the chart at least. I am hoping that the .net version (which i havent experimented with) will have the ability to easily store and fetch parameters for optimizations.

tony95
Posts: 20
Joined: 10 Sep 2012
Has thanked: 4 times
Been thanked: 2 times

Re: Best computer for Genetic Algo optimization

Postby tony95 » 16 Sep 2012

I would get a LGA2011 dual processor board and use two Intel Core i7-3930K on it. If you have money to burn then you can go with the xeons but you won't get much more out of them.
http://www.cpubenchmark.net/high_end_cpus.html

I can't imagine you would need more than 64GB of memory. Can you estimate your need based on what you are using now? Use RAID on a bunch of SSDs and you will get super fast disk access.

I believe the K processors can be overclocked as well, but it is not something I have ever felt the need to do.
These users thanked the author tony95 for the post:
KhaosTrader

KhaosTrader
Posts: 186
Joined: 10 May 2012
Has thanked: 14 times
Been thanked: 10 times

Re: Best computer for Genetic Algo optimization

Postby KhaosTrader » 16 Sep 2012

Thanks Tony,

Do you know where I could buy that motherboard and chip online? I live overseas so I would need to have it shipped to me...

Thanks

tony95
Posts: 20
Joined: 10 Sep 2012
Has thanked: 4 times
Been thanked: 2 times

Re: Best computer for Genetic Algo optimization

Postby tony95 » 16 Sep 2012

Are you in the UK?

KhaosTrader
Posts: 186
Joined: 10 May 2012
Has thanked: 14 times
Been thanked: 10 times

Re: Best computer for Genetic Algo optimization

Postby KhaosTrader » 16 Sep 2012

I am in the Philippines...

User avatar
TJ
Posts: 6584
Joined: 29 Aug 2006
Location: Global Citizen
Has thanked: 970 times
Been thanked: 1907 times

Re: Best computer for Genetic Algo optimization

Postby TJ » 16 Sep 2012

KhaosTrader wrote:What is the best computer type for Genetic Algo (GA) optimization?
...

Also are you guys going to implement grid computing architecture in the future where I can have multiple computers crunch on a GA optimization?

....

Grid computing is not a plug and play solution, it requires a middleware to divide and apportion pieces of a program among several computers.

For more information:
see post #14, FAQ on Multi-Core, Multi-CPU, CUDA, Grid Computing, etc.,
last paragraph
viewtopic.php?f=16&t=6845
These users thanked the author TJ for the post:
JoshM

tony95
Posts: 20
Joined: 10 Sep 2012
Has thanked: 4 times
Been thanked: 2 times

Re: Best computer for Genetic Algo optimization

Postby tony95 » 16 Sep 2012

After looking closer, only the XEONs are going to run on the dual socket boards. It is confusing because they are both LGA 2011. So, I would still go with the dual socket if you can afford it. With this option you can go with 128GB memory, so you will be in good shape there.

Here are some of the boards you can research more.

http://www.newegg.com/Product/ProductList.aspx?Submit=ENE&DEPA=0&Order=BESTMATCH&N=100006654&isNodeId=1&Description=dual+lga+2011

You should be able to order from Amazon or Newegg.
These users thanked the author tony95 for the post:
TJ

KhaosTrader
Posts: 186
Joined: 10 May 2012
Has thanked: 14 times
Been thanked: 10 times

Re: Best computer for Genetic Algo optimization

Postby KhaosTrader » 16 Sep 2012

Ok i have been searching on Asus they have so many boards.. lol.

Questions for Tony:

1) which particular board do your recommend?

2) Are you saying I must get XEONs for dual processor?

tony95
Posts: 20
Joined: 10 Sep 2012
Has thanked: 4 times
Been thanked: 2 times

Re: Best computer for Genetic Algo optimization

Postby tony95 » 16 Sep 2012

Yeah, you will have to use a XEON processor for dual socket boards. Apparently, the i7's are missing the feature that allows them to work with another processor.

People seem to like this one. But you will have to find someone who will ship it to you.
http://www.newegg.com/Product/Product.a ... 6813182343

KhaosTrader
Posts: 186
Joined: 10 May 2012
Has thanked: 14 times
Been thanked: 10 times

Re: Best computer for Genetic Algo optimization

Postby KhaosTrader » 16 Sep 2012

Thanks Tony, appreciate your help on this...

I wonder how much difference 1 cpu vs 2 cpu computer is..

If I go with 1 cpu, i can go with the i7 chip u are talking about

If I go XEON dual, the cost is perhaps 5x or 6x.

tony95
Posts: 20
Joined: 10 Sep 2012
Has thanked: 4 times
Been thanked: 2 times

Re: Best computer for Genetic Algo optimization

Postby tony95 » 16 Sep 2012

I wouldn't take my opinion on which board. You should read the reviews on the first few boards and see which you think will meet your needs. I think the board I mentioned was rated well because it is economical. You may not care so much about that.

You may prefer the ASUS Z9PE-D8 WS which I think has more features but is more expensive.

tony95
Posts: 20
Joined: 10 Sep 2012
Has thanked: 4 times
Been thanked: 2 times

Re: Best computer for Genetic Algo optimization

Postby tony95 » 16 Sep 2012

You should get twice the processing power from the dual cpu, but the software you are using has to be highly multi threaded. I am guessing that the optimizers are. You could build two i7 machines, but then you have to buy another MC license which blows your savings. You'll have to decided how much power you really need to accomplish your goal.

KhaosTrader
Posts: 186
Joined: 10 May 2012
Has thanked: 14 times
Been thanked: 10 times

Re: Best computer for Genetic Algo optimization

Postby KhaosTrader » 16 Sep 2012

Yeah I already have 2 licenses of .net version and 2 licenses of the easylanguage version, if i go that route, I would need one more of each, as I plan to move some of my easylanguage to .net in the coming months...

The good reason tho to have 2 computers also would be the idea that technology keeps improving, and I can replace the computers every 18 months or so, if i spent 7k usd on an xeon server that would make it harder to replace as fast since the investment was larger.

Realize also that since I live overseas, getting high end equipment causes me to pay a lot more for shipping, duties and taxes... Which is an issue..

So the gameplan would be to buy two "optimization computers" and have them run and build models and then i can save them to my LAN storage device and then my trading computer can just grab the updated charts for trading.

Thanks for your help Tony, I appreciate it.

I assume you are running a lot of optimizations also, what setup do you use?

KhaosTrader
Posts: 186
Joined: 10 May 2012
Has thanked: 14 times
Been thanked: 10 times

Re: Best computer for Genetic Algo optimization

Postby KhaosTrader » 16 Sep 2012

I did some more research and it actually looks like I can probably get the parts in malaysia or thailand and then bring it over personally, that would then make it more effective to go with a XEON server...

So I need tech support to answer if indeed the genetic algo will utilize a dual XEON processor and how much memory will it utilize, in otherwords should i get 64gb or 128 gb RAM

User avatar
Henry MultiСharts
Posts: 8504
Joined: 25 Aug 2011
Has thanked: 1213 times
Been thanked: 2732 times

Re: Best computer for Genetic Algo optimization

Postby Henry MultiСharts » 17 Sep 2012

KhaosTrader wrote:So I need tech support to answer if indeed the genetic algo will utilize a dual XEON processor and how much memory will it utilize, in otherwords should i get 64gb or 128 gb RAM


KhaosTrader, it depends on the calculation you will perform. MultiCharts will try to use all of the cores on your PC and take as much memory as it needs.

KhaosTrader
Posts: 186
Joined: 10 May 2012
Has thanked: 14 times
Been thanked: 10 times

Re: Best computer for Genetic Algo optimization

Postby KhaosTrader » 17 Sep 2012

will it support hyperthreading?

also 64gb vs 128 gb, is that necessary to go to 128gb? its a lot of memory even at 64gb

User avatar
Henry MultiСharts
Posts: 8504
Joined: 25 Aug 2011
Has thanked: 1213 times
Been thanked: 2732 times

Re: Best computer for Genetic Algo optimization  [SOLVED]

Postby Henry MultiСharts » 17 Sep 2012

KhaosTrader wrote:will it support hyperthreading?

Hyperthreading is supported, though optimization time with HT and without it is the same as HT is core virtualization.

KhaosTrader wrote:also 64gb vs 128 gb, is that necessary to go to 128gb? its a lot of memory even at 64gb
It is up to you to decide.
These users thanked the author Henry MultiСharts for the post:
KhaosTrader

imoneyfish
Posts: 37
Joined: 26 Oct 2011
Has thanked: 6 times
Been thanked: 2 times

Re: Best computer for Genetic Algo optimization

Postby imoneyfish » 17 Feb 2013

http://screencast.com/t/w7V6kSLVddem


http://screencast.com/users/imoneyfish/folders/Default/media/d5133f50-3d5b-4395-9977-0263d793e42c

Image


Trust me. Not enough. I am upgrading to 20 cpus minimum. Not find how to do that yet. Any suggestions? Thanks.

User avatar
ABC
Posts: 524
Joined: 16 Dec 2006
Has thanked: 111 times
Been thanked: 295 times
Contact:

Re: Best computer for Genetic Algo optimization

Postby ABC » 18 Feb 2013

From experience I know that there is one topic people tend to overlook when it comes to optimization times. Don't get me wrong, I am not saying this is the case here, but as the topic is of interest for many, why not write it here.
I have found the way code is written to have a huge impact on optimization or calculation times. In other words it makes sense to specifically optimize your code for performance and you can do that easily with free performance timers that are around. So you can go through the code step by step and see what effect a small change has.
To give you an idea, I have seen calculations times being ten times faster for the same system after some basic changes to the code (with the same outcome and on the same hardware of course). While this might be the exception, it's still possible to cut the optimization times by half (or more) for many systems, with just optimizing the code.
So besides speeding up the hardware, it's always wise to speed up the "software", too. Only spending big bucks on hardware and overlooking the other part of the equation means you don't get your money's worth to an extend possible.

Regards,
ABC
These users thanked the author ABC for the post (total 2):
TJHenry MultiСharts

txls
Posts: 76
Joined: 17 Nov 2010
Has thanked: 36 times
Been thanked: 24 times

Re: Best computer for Genetic Algo optimization

Postby txls » 19 Feb 2013

ABC wrote:From experience I know


I'm guessing you are mainly referring to MC.NET in your post. Or is it the same with the regular MC in your experience?

User avatar
ABC
Posts: 524
Joined: 16 Dec 2006
Has thanked: 111 times
Been thanked: 295 times
Contact:

Re: Best computer for Genetic Algo optimization

Postby ABC » 19 Feb 2013

txls wrote:
ABC wrote:From experience I know


I'm guessing you are mainly referring to MC.NET in your post. Or is it the same with the regular MC in your experience?


I was referring to the regular MC, but I am sure this goes for MC.NET as well.

txls
Posts: 76
Joined: 17 Nov 2010
Has thanked: 36 times
Been thanked: 24 times

Re: Best computer for Genetic Algo optimization

Postby txls » 19 Feb 2013

I'm curious, how would you assess regular MC's PL study performance using a performance timer?

User avatar
ABC
Posts: 524
Joined: 16 Dec 2006
Has thanked: 111 times
Been thanked: 295 times
Contact:

Re: Best computer for Genetic Algo optimization

Postby ABC » 19 Feb 2013

txls wrote:I'm curious, how would you assess regular MC's PL study performance using a performance timer?


For example there is a tool called EL Performance Timer, which was written by Bamboo for TS originally (if I recall it correctly). It will output the calculation times.

hilbert
Posts: 222
Joined: 17 Aug 2011
Has thanked: 76 times
Been thanked: 64 times

Re: Best computer for Genetic Algo optimization

Postby hilbert » 19 Feb 2013

ABC wrote:
txls wrote:I'm curious, how would you assess regular MC's PL study performance using a performance timer?


For example there is a tool called EL Performance Timer, which was written by Bamboo for TS originally (if I recall it correctly). It will output the calculation times.


Hello,

I looked around but couldn't find the tool by Bamboo. Can you please send it over to me through email? Thanks very much.

-Hilbert

User avatar
JoshM
Posts: 2089
Joined: 20 May 2011
Location: The Netherlands
Has thanked: 1525 times
Been thanked: 1485 times
Contact:

Re: Best computer for Genetic Algo optimization

Postby JoshM » 06 Mar 2013

Would Amazon EC2 be a viable alternative for backtesting and optimizing in MultiCharts/MultiCharts .NET instead of buying a new high-end desktop pc every year?

Given the (relatively) lower cost of computing power through EC2 and no need to invest in (maintenance of) hardware, this might be cheaper. An additional benefit is that you can keep working on your pc while the EC2 pc performs the backtest.

However, I don't have experience with EC2 so I'd like to hear your opinions on this. Would EC2 be a good alternative, or is the use of a desktop pc much better/handier?

imoneyfish
Posts: 37
Joined: 26 Oct 2011
Has thanked: 6 times
Been thanked: 2 times

Re: Best computer for Genetic Algo optimization

Postby imoneyfish » 06 Mar 2013

JoshM wrote:Would Amazon EC2 be a viable alternative for backtesting and optimizing in MultiCharts/MultiCharts .NET instead of buying a new high-end desktop pc every year?

Given the (relatively) lower cost of computing power through EC2 and no need to invest in (maintenance of) hardware, this might be cheaper. An additional benefit is that you can keep working on your pc while the EC2 pc performs the backtest.

However, I don't have experience with EC2 so I'd like to hear your opinions on this. Would EC2 be a good alternative, or is the use of a desktop pc much better/handier?



Hello Jos, I tried EC2 before. Not too bad if you only do optimization occasionally. I think you can stop the instance and reopen it and you files everything will still be there. I only tried once though. MC really need to support GPU, otherwise optimization just too too slow. I saw on YouTube a guy seems can link MatLab( Which can use the GPU capacity ) with MC. Anyone interested may worth a look.
These users thanked the author imoneyfish for the post:
JoshM

TW
Posts: 29
Joined: 14 Dec 2009
Has thanked: 4 times

Re: Best computer for Genetic Algo optimization

Postby TW » 28 Jun 2017

Hello Jos, I tried EC2 before. Not too bad if you only do optimization occasionally. I think you can stop the instance and reopen it and you files everything will still be there. I only tried once though. MC really need to support GPU, otherwise optimization just too too slow. I saw on YouTube a guy seems can link MatLab( Which can use the GPU capacity ) with MC. Anyone interested may worth a look.[/quote]

I really hope that MC could support GPU to improve the efficiency of optimization. Hope MC team could think about my suggestion and make it in plan.

User avatar
Henry MultiСharts
Posts: 8504
Joined: 25 Aug 2011
Has thanked: 1213 times
Been thanked: 2732 times

Re: Best computer for Genetic Algo optimization

Postby Henry MultiСharts » 28 Jun 2017

TW wrote:I really hope that MC could support GPU to improve the efficiency of optimization. Hope MC team could think about my suggestion and make it in plan.
TW,

This is not something that is in our to-do list at the moment since it requires rebuilding the platform architecture from scratch.


Return to “MultiCharts”