Best computer for Genetic Algo optimization [SOLVED]
-
- Posts: 186
- Joined: 10 May 2012
- Has thanked: 14 times
- Been thanked: 11 times
Best computer for Genetic Algo optimization
What is the best computer type for Genetic Algo (GA) optimization?
Should I go with XEON processor with 128 gig memory? or just a standard hex core intel processor with 64 gig memory?
Also are you guys going to implement grid computing architecture in the future where I can have multiple computers crunch on a GA optimization?
Currently I bought 2 licenses of MC (and I have two MC.nets also) -- one license is for me to trade with , and the other one just crunches optimizations throughout the day. I really would like to make my optimizations work faster so that I can test a wider parameter space.
Also I really would like it if i could save the ranges of optimizations, rather than have to enter them every time i restart mc, they should be saved in the chart at least. I am hoping that the .net version (which i havent experimented with) will have the ability to easily store and fetch parameters for optimizations.
Should I go with XEON processor with 128 gig memory? or just a standard hex core intel processor with 64 gig memory?
Also are you guys going to implement grid computing architecture in the future where I can have multiple computers crunch on a GA optimization?
Currently I bought 2 licenses of MC (and I have two MC.nets also) -- one license is for me to trade with , and the other one just crunches optimizations throughout the day. I really would like to make my optimizations work faster so that I can test a wider parameter space.
Also I really would like it if i could save the ranges of optimizations, rather than have to enter them every time i restart mc, they should be saved in the chart at least. I am hoping that the .net version (which i havent experimented with) will have the ability to easily store and fetch parameters for optimizations.
Re: Best computer for Genetic Algo optimization
I would get a LGA2011 dual processor board and use two Intel Core i7-3930K on it. If you have money to burn then you can go with the xeons but you won't get much more out of them.
http://www.cpubenchmark.net/high_end_cpus.html
I can't imagine you would need more than 64GB of memory. Can you estimate your need based on what you are using now? Use RAID on a bunch of SSDs and you will get super fast disk access.
I believe the K processors can be overclocked as well, but it is not something I have ever felt the need to do.
http://www.cpubenchmark.net/high_end_cpus.html
I can't imagine you would need more than 64GB of memory. Can you estimate your need based on what you are using now? Use RAID on a bunch of SSDs and you will get super fast disk access.
I believe the K processors can be overclocked as well, but it is not something I have ever felt the need to do.
-
- Posts: 186
- Joined: 10 May 2012
- Has thanked: 14 times
- Been thanked: 11 times
Re: Best computer for Genetic Algo optimization
Thanks Tony,
Do you know where I could buy that motherboard and chip online? I live overseas so I would need to have it shipped to me...
Thanks
Do you know where I could buy that motherboard and chip online? I live overseas so I would need to have it shipped to me...
Thanks
-
- Posts: 186
- Joined: 10 May 2012
- Has thanked: 14 times
- Been thanked: 11 times
- TJ
- Posts: 7744
- Joined: 29 Aug 2006
- Location: Global Citizen
- Has thanked: 1033 times
- Been thanked: 2224 times
Re: Best computer for Genetic Algo optimization
Grid computing is not a plug and play solution, it requires a middleware to divide and apportion pieces of a program among several computers.What is the best computer type for Genetic Algo (GA) optimization?
...
Also are you guys going to implement grid computing architecture in the future where I can have multiple computers crunch on a GA optimization?
....
For more information:
see post #14, FAQ on Multi-Core, Multi-CPU, CUDA, Grid Computing, etc.,
last paragraph
viewtopic.php?f=16&t=6845
Re: Best computer for Genetic Algo optimization
After looking closer, only the XEONs are going to run on the dual socket boards. It is confusing because they are both LGA 2011. So, I would still go with the dual socket if you can afford it. With this option you can go with 128GB memory, so you will be in good shape there.
Here are some of the boards you can research more.
http://www.newegg.com/Product/ProductLi ... l+lga+2011
You should be able to order from Amazon or Newegg.
Here are some of the boards you can research more.
http://www.newegg.com/Product/ProductLi ... l+lga+2011
You should be able to order from Amazon or Newegg.
-
- Posts: 186
- Joined: 10 May 2012
- Has thanked: 14 times
- Been thanked: 11 times
Re: Best computer for Genetic Algo optimization
Ok i have been searching on Asus they have so many boards.. lol.
Questions for Tony:
1) which particular board do your recommend?
2) Are you saying I must get XEONs for dual processor?
Questions for Tony:
1) which particular board do your recommend?
2) Are you saying I must get XEONs for dual processor?
Re: Best computer for Genetic Algo optimization
Yeah, you will have to use a XEON processor for dual socket boards. Apparently, the i7's are missing the feature that allows them to work with another processor.
People seem to like this one. But you will have to find someone who will ship it to you.
http://www.newegg.com/Product/Product.a ... 6813182343
People seem to like this one. But you will have to find someone who will ship it to you.
http://www.newegg.com/Product/Product.a ... 6813182343
-
- Posts: 186
- Joined: 10 May 2012
- Has thanked: 14 times
- Been thanked: 11 times
Re: Best computer for Genetic Algo optimization
Thanks Tony, appreciate your help on this...
I wonder how much difference 1 cpu vs 2 cpu computer is..
If I go with 1 cpu, i can go with the i7 chip u are talking about
If I go XEON dual, the cost is perhaps 5x or 6x.
I wonder how much difference 1 cpu vs 2 cpu computer is..
If I go with 1 cpu, i can go with the i7 chip u are talking about
If I go XEON dual, the cost is perhaps 5x or 6x.
Re: Best computer for Genetic Algo optimization
I wouldn't take my opinion on which board. You should read the reviews on the first few boards and see which you think will meet your needs. I think the board I mentioned was rated well because it is economical. You may not care so much about that.
You may prefer the ASUS Z9PE-D8 WS which I think has more features but is more expensive.
You may prefer the ASUS Z9PE-D8 WS which I think has more features but is more expensive.
Re: Best computer for Genetic Algo optimization
You should get twice the processing power from the dual cpu, but the software you are using has to be highly multi threaded. I am guessing that the optimizers are. You could build two i7 machines, but then you have to buy another MC license which blows your savings. You'll have to decided how much power you really need to accomplish your goal.
-
- Posts: 186
- Joined: 10 May 2012
- Has thanked: 14 times
- Been thanked: 11 times
Re: Best computer for Genetic Algo optimization
Yeah I already have 2 licenses of .net version and 2 licenses of the easylanguage version, if i go that route, I would need one more of each, as I plan to move some of my easylanguage to .net in the coming months...
The good reason tho to have 2 computers also would be the idea that technology keeps improving, and I can replace the computers every 18 months or so, if i spent 7k usd on an xeon server that would make it harder to replace as fast since the investment was larger.
Realize also that since I live overseas, getting high end equipment causes me to pay a lot more for shipping, duties and taxes... Which is an issue..
So the gameplan would be to buy two "optimization computers" and have them run and build models and then i can save them to my LAN storage device and then my trading computer can just grab the updated charts for trading.
Thanks for your help Tony, I appreciate it.
I assume you are running a lot of optimizations also, what setup do you use?
The good reason tho to have 2 computers also would be the idea that technology keeps improving, and I can replace the computers every 18 months or so, if i spent 7k usd on an xeon server that would make it harder to replace as fast since the investment was larger.
Realize also that since I live overseas, getting high end equipment causes me to pay a lot more for shipping, duties and taxes... Which is an issue..
So the gameplan would be to buy two "optimization computers" and have them run and build models and then i can save them to my LAN storage device and then my trading computer can just grab the updated charts for trading.
Thanks for your help Tony, I appreciate it.
I assume you are running a lot of optimizations also, what setup do you use?
-
- Posts: 186
- Joined: 10 May 2012
- Has thanked: 14 times
- Been thanked: 11 times
Re: Best computer for Genetic Algo optimization
I did some more research and it actually looks like I can probably get the parts in malaysia or thailand and then bring it over personally, that would then make it more effective to go with a XEON server...
So I need tech support to answer if indeed the genetic algo will utilize a dual XEON processor and how much memory will it utilize, in otherwords should i get 64gb or 128 gb RAM
So I need tech support to answer if indeed the genetic algo will utilize a dual XEON processor and how much memory will it utilize, in otherwords should i get 64gb or 128 gb RAM
- Henry MultiСharts
- Posts: 9165
- Joined: 25 Aug 2011
- Has thanked: 1264 times
- Been thanked: 2957 times
Re: Best computer for Genetic Algo optimization
KhaosTrader, it depends on the calculation you will perform. MultiCharts will try to use all of the cores on your PC and take as much memory as it needs.So I need tech support to answer if indeed the genetic algo will utilize a dual XEON processor and how much memory will it utilize, in otherwords should i get 64gb or 128 gb RAM
-
- Posts: 186
- Joined: 10 May 2012
- Has thanked: 14 times
- Been thanked: 11 times
Re: Best computer for Genetic Algo optimization
will it support hyperthreading?
also 64gb vs 128 gb, is that necessary to go to 128gb? its a lot of memory even at 64gb
also 64gb vs 128 gb, is that necessary to go to 128gb? its a lot of memory even at 64gb
- Henry MultiСharts
- Posts: 9165
- Joined: 25 Aug 2011
- Has thanked: 1264 times
- Been thanked: 2957 times
Re: Best computer for Genetic Algo optimization [SOLVED]
Hyperthreading is supported, though optimization time with HT and without it is the same as HT is core virtualization.will it support hyperthreading?
It is up to you to decide.also 64gb vs 128 gb, is that necessary to go to 128gb? its a lot of memory even at 64gb
-
- Posts: 38
- Joined: 26 Oct 2011
- Has thanked: 6 times
- Been thanked: 2 times
Re: Best computer for Genetic Algo optimization
http://screencast.com/t/w7V6kSLVddem
http://screencast.com/users/imoneyfish/ ... 63d793e42c
Trust me. Not enough. I am upgrading to 20 cpus minimum. Not find how to do that yet. Any suggestions? Thanks.
http://screencast.com/users/imoneyfish/ ... 63d793e42c
Trust me. Not enough. I am upgrading to 20 cpus minimum. Not find how to do that yet. Any suggestions? Thanks.
- ABC
- Posts: 723
- Joined: 16 Dec 2006
- Location: www.abctradinggroup.com
- Has thanked: 125 times
- Been thanked: 410 times
- Contact:
Re: Best computer for Genetic Algo optimization
From experience I know that there is one topic people tend to overlook when it comes to optimization times. Don't get me wrong, I am not saying this is the case here, but as the topic is of interest for many, why not write it here.
I have found the way code is written to have a huge impact on optimization or calculation times. In other words it makes sense to specifically optimize your code for performance and you can do that easily with free performance timers that are around. So you can go through the code step by step and see what effect a small change has.
To give you an idea, I have seen calculations times being ten times faster for the same system after some basic changes to the code (with the same outcome and on the same hardware of course). While this might be the exception, it's still possible to cut the optimization times by half (or more) for many systems, with just optimizing the code.
So besides speeding up the hardware, it's always wise to speed up the "software", too. Only spending big bucks on hardware and overlooking the other part of the equation means you don't get your money's worth to an extend possible.
Regards,
ABC
I have found the way code is written to have a huge impact on optimization or calculation times. In other words it makes sense to specifically optimize your code for performance and you can do that easily with free performance timers that are around. So you can go through the code step by step and see what effect a small change has.
To give you an idea, I have seen calculations times being ten times faster for the same system after some basic changes to the code (with the same outcome and on the same hardware of course). While this might be the exception, it's still possible to cut the optimization times by half (or more) for many systems, with just optimizing the code.
So besides speeding up the hardware, it's always wise to speed up the "software", too. Only spending big bucks on hardware and overlooking the other part of the equation means you don't get your money's worth to an extend possible.
Regards,
ABC
- ABC
- Posts: 723
- Joined: 16 Dec 2006
- Location: www.abctradinggroup.com
- Has thanked: 125 times
- Been thanked: 410 times
- Contact:
Re: Best computer for Genetic Algo optimization
I was referring to the regular MC, but I am sure this goes for MC.NET as well.I'm guessing you are mainly referring to MC.NET in your post. Or is it the same with the regular MC in your experience?From experience I know
- ABC
- Posts: 723
- Joined: 16 Dec 2006
- Location: www.abctradinggroup.com
- Has thanked: 125 times
- Been thanked: 410 times
- Contact:
Re: Best computer for Genetic Algo optimization
For example there is a tool called EL Performance Timer, which was written by Bamboo for TS originally (if I recall it correctly). It will output the calculation times.I'm curious, how would you assess regular MC's PL study performance using a performance timer?
Re: Best computer for Genetic Algo optimization
Hello,For example there is a tool called EL Performance Timer, which was written by Bamboo for TS originally (if I recall it correctly). It will output the calculation times.I'm curious, how would you assess regular MC's PL study performance using a performance timer?
I looked around but couldn't find the tool by Bamboo. Can you please send it over to me through email? Thanks very much.
-Hilbert
- JoshM
- Posts: 2195
- Joined: 20 May 2011
- Location: The Netherlands
- Has thanked: 1544 times
- Been thanked: 1565 times
- Contact:
Re: Best computer for Genetic Algo optimization
Would Amazon EC2 be a viable alternative for backtesting and optimizing in MultiCharts/MultiCharts .NET instead of buying a new high-end desktop pc every year?
Given the (relatively) lower cost of computing power through EC2 and no need to invest in (maintenance of) hardware, this might be cheaper. An additional benefit is that you can keep working on your pc while the EC2 pc performs the backtest.
However, I don't have experience with EC2 so I'd like to hear your opinions on this. Would EC2 be a good alternative, or is the use of a desktop pc much better/handier?
Given the (relatively) lower cost of computing power through EC2 and no need to invest in (maintenance of) hardware, this might be cheaper. An additional benefit is that you can keep working on your pc while the EC2 pc performs the backtest.
However, I don't have experience with EC2 so I'd like to hear your opinions on this. Would EC2 be a good alternative, or is the use of a desktop pc much better/handier?
-
- Posts: 38
- Joined: 26 Oct 2011
- Has thanked: 6 times
- Been thanked: 2 times
Re: Best computer for Genetic Algo optimization
Would Amazon EC2 be a viable alternative for backtesting and optimizing in MultiCharts/MultiCharts .NET instead of buying a new high-end desktop pc every year?
Given the (relatively) lower cost of computing power through EC2 and no need to invest in (maintenance of) hardware, this might be cheaper. An additional benefit is that you can keep working on your pc while the EC2 pc performs the backtest.
However, I don't have experience with EC2 so I'd like to hear your opinions on this. Would EC2 be a good alternative, or is the use of a desktop pc much better/handier?
Hello Jos, I tried EC2 before. Not too bad if you only do optimization occasionally. I think you can stop the instance and reopen it and you files everything will still be there. I only tried once though. MC really need to support GPU, otherwise optimization just too too slow. I saw on YouTube a guy seems can link MatLab( Which can use the GPU capacity ) with MC. Anyone interested may worth a look.
Re: Best computer for Genetic Algo optimization
Hello Jos, I tried EC2 before. Not too bad if you only do optimization occasionally. I think you can stop the instance and reopen it and you files everything will still be there. I only tried once though. MC really need to support GPU, otherwise optimization just too too slow. I saw on YouTube a guy seems can link MatLab( Which can use the GPU capacity ) with MC. Anyone interested may worth a look.[/quote]
I really hope that MC could support GPU to improve the efficiency of optimization. Hope MC team could think about my suggestion and make it in plan.
I really hope that MC could support GPU to improve the efficiency of optimization. Hope MC team could think about my suggestion and make it in plan.
- Henry MultiСharts
- Posts: 9165
- Joined: 25 Aug 2011
- Has thanked: 1264 times
- Been thanked: 2957 times
Re: Best computer for Genetic Algo optimization
TW,I really hope that MC could support GPU to improve the efficiency of optimization. Hope MC team could think about my suggestion and make it in plan.
This is not something that is in our to-do list at the moment since it requires rebuilding the platform architecture from scratch.