Hello everyone,
I am just about to start a series of serious optimizations. My little laptop with Intel T7400 is too slow. Please see a print screen attached. It takes 61 hours to calculate one strategy on 3.5 year data (20 minute compression). I need to calculate at least 20 such strategies.
Can you advise me from your own experience what computer would be best for backtesting and optimization? I am not asking for an advice on a personal computer or a workstation - these will be too slow. I am asking for an advice on a server of high computing power.
I would appreciate any help on a specification of such a server (type of processors, number of processors (4/8/16/32?), operating system. If you were using servers for backtesting and optimization and you noticed a boost in performance over a personal computer, please let me know the name of this server. If you were using 2 servers and noticed that one overperformed the other, please tell me
Regards
Best machine for optimization (processor type, OS, others)?
Best machine for optimization (processor type, OS, others)?
- Attachments
-
- optimization.jpg
- (102.29 KiB) Downloaded 1486 times
Last edited by Tresor on 01 Jun 2008, edited 1 time in total.
- TJ
- Posts: 7752
- Joined: 29 Aug 2006
- Location: Global Citizen
- Has thanked: 1033 times
- Been thanked: 2228 times
you can take a look at this test run:
http://www.elitetrader.com/vb/showthrea ... ost1849141
It was done on a Intel Q6600 quadcore processor.
You can see in the Task Manager that MultiCharts utilized all 4 CPUs to the max !
http://www.elitetrader.com/vb/showthrea ... ost1849141
It was done on a Intel Q6600 quadcore processor.
You can see in the Task Manager that MultiCharts utilized all 4 CPUs to the max !
Hello TJ,you can take a look at this test run:
http://www.elitetrader.com/vb/showthrea ... ost1849141
It was done on a Intel Q6600 quadcore processor.
You can see in the Task Manager that MultiCharts utilized all 4 CPUs to the max !
Many thanks for the link. The machine Tums built is made of one processor. And from what I understand all four cores of this processor are 99% utilised while optimizing. Great news!
Do you think that when I build a machine of four Intel Q6600 processors the optimization that MC will be doing will be four times faster than the optimization run just on one Intel Q6600?
Regards
- Marina Pashkova
- Posts: 2758
- Joined: 27 Jul 2007
- Marina Pashkova
- Posts: 2758
- Joined: 27 Jul 2007
Hi Marina,Hi Tresor,
Do you have any particular questions regarding optimization on multi-core machines?
Regards.
There's is one question that I have had in my mind for a few days. Namely, I contacted Microsoft Support and a guy told me that even if I buy a machine with 4 CPUs, then XP Proffesional and Vista will only use maximum of 2 CPUs and will be blind for the remining 2 CPUs and will prevent any other application to utilise more than 2 CPUs.
In other words, I should buy a machine with no more than 2 CPUs. Can you confirm what this guy told me?
Regards
- Marina Pashkova
- Posts: 2758
- Joined: 27 Jul 2007
Hi Tresor,
According to this document http://www.microsoft.com/licensing/high ... icore.mspx what you're saying is true: more than 2 processors can't be used in XP and Vista.
To be honest, we're not sure if it really won't work. Actually, all 4 cores might be used in optimization on XP. I would recommend trying to run optimization on a quad-core machine and check the CPU load.
Or you could purchase Windows Server - it'll use all the available cores for optimization for sure.
Regards.
According to this document http://www.microsoft.com/licensing/high ... icore.mspx what you're saying is true: more than 2 processors can't be used in XP and Vista.
To be honest, we're not sure if it really won't work. Actually, all 4 cores might be used in optimization on XP. I would recommend trying to run optimization on a quad-core machine and check the CPU load.
Or you could purchase Windows Server - it'll use all the available cores for optimization for sure.
Regards.
Hi Marina,Hi Tresor,
According to this document http://www.microsoft.com/licensing/high ... icore.mspx what you're saying is true: more than 2 processors can't be used in XP and Vista.
To be honest, we're not sure if it really won't work. Actually, all 4 cores might be used in optimization on XP. I would recommend trying to run optimization on a quad-core machine and check the CPU load.
Or you could purchase Windows Server - it'll use all the available cores for optimization for sure.
Regards.
That would make things different if I could use Windows Server as my operating system Having read this system requirements http://www.tssupport.com/multicharts/requirements/ I just thought that only operating systems that MC can work with are XP and Vista.
Regards
- Marina Pashkova
- Posts: 2758
- Joined: 27 Jul 2007
Hi all,
I have run 3 tests of VPCI strategy on on 3.5 year data, 16 minute time bars, with the same optimizable inputs. The results were as follows
1. Fujitsu-siemens laptop T4215 with 1 x Intel T7400, 2.16 GHz, 2GB RAM, XP 32 bit, time: 14:02:52
2. HP DL 140 server with 1 x Intel Xeon E5130, 2GB RAM, XP 64 bit, time: 10:37:05
3. Dell Power Edge 2950 server with 2 x Intel Xeon E5430, 2.55 GHz, 8GB RAM, Windows 2003 Server, time: 2:11:26
I attach the print screen of the last test. It is confirmed Portfolio Backtester can run on Windows Server, all 8 cores are 97% used
This test shows that the performance of Portfolio Backtester on 8 cores with Windows Server installed and 4 x higher RAM is 6.6 times quicker than on 2 cores of my laptop. Good news.
Now, did any of guys tried to optimize on a machine with more than 2 CPUs? If yes, then how the performance was increased?
Regards
BTW, Marina can you also add Windows Server operating system to System Requirements on your website. Maybe some of institutional customers or heavy optimizers might find this info useful.
I have run 3 tests of VPCI strategy on on 3.5 year data, 16 minute time bars, with the same optimizable inputs. The results were as follows
1. Fujitsu-siemens laptop T4215 with 1 x Intel T7400, 2.16 GHz, 2GB RAM, XP 32 bit, time: 14:02:52
2. HP DL 140 server with 1 x Intel Xeon E5130, 2GB RAM, XP 64 bit, time: 10:37:05
3. Dell Power Edge 2950 server with 2 x Intel Xeon E5430, 2.55 GHz, 8GB RAM, Windows 2003 Server, time: 2:11:26
I attach the print screen of the last test. It is confirmed Portfolio Backtester can run on Windows Server, all 8 cores are 97% used
This test shows that the performance of Portfolio Backtester on 8 cores with Windows Server installed and 4 x higher RAM is 6.6 times quicker than on 2 cores of my laptop. Good news.
Now, did any of guys tried to optimize on a machine with more than 2 CPUs? If yes, then how the performance was increased?
Regards
BTW, Marina can you also add Windows Server operating system to System Requirements on your website. Maybe some of institutional customers or heavy optimizers might find this info useful.
- Attachments
-
- Dell 8 core optimization.jpg
- (124.53 KiB) Downloaded 1477 times
- TJ
- Posts: 7752
- Joined: 29 Aug 2006
- Location: Global Citizen
- Has thanked: 1033 times
- Been thanked: 2228 times
Tresor:
Thank you for sharing your test.
Good to see how 2x CPU produces better than 3x result.
FYI, on the last test, only 2.5GB of RAM was used. Therefore the additional RAM beyond 4GB was not a factor.
note: XP can utilize 3GB RAM. Most people install 4GB in their computers these days.
note: if the program requires 2.5GB, and your computer must swap memory because it only has 2GB, then the penalty in swapping is very high.
I would be interested to see how your second test would perform, if the computer has 4GB of memory. I think it will take less than half the time it took.
best regards
TJ
Thank you for sharing your test.
Good to see how 2x CPU produces better than 3x result.
FYI, on the last test, only 2.5GB of RAM was used. Therefore the additional RAM beyond 4GB was not a factor.
note: XP can utilize 3GB RAM. Most people install 4GB in their computers these days.
note: if the program requires 2.5GB, and your computer must swap memory because it only has 2GB, then the penalty in swapping is very high.
I would be interested to see how your second test would perform, if the computer has 4GB of memory. I think it will take less than half the time it took.
best regards
TJ
Hi TJ,Tresor:
Thank you for sharing your test.
Good to see how 2x CPU produces better than 3x result.
FYI, on the last test, only 2.5GB of RAM was used. Therefore the additional RAM beyond 4GB was not a factor.
note: XP can utilize 3GB RAM. Most people install 4GB in their computers these days.
note: if the program requires 2.5GB, and your computer must swap memory because it only has 2GB, then the penalty in swapping is very high.
I would be interested to see how your second test would perform, if the computer has 4GB of memory. I think it will take less than half the time it took.
best regards
TJ
It really seems that buying a computer with more than 4 GB RAM would be throwing away the money as Backtester uses 2.5 GB. Thanks for this advice.
I already am searching for a machine in my vicinity that has 4 (or more) CPUs x 4 cores that I can install MC on and put to test. The ctach is here though that such machines big have Unix installed, not Windows. So my search will be a rather lenghthy one.
As soons as I get my hands on such a machine I will post the results.
Regards
- TJ
- Posts: 7752
- Joined: 29 Aug 2006
- Location: Global Citizen
- Has thanked: 1033 times
- Been thanked: 2228 times
if you can "borrow" a powerful Intel machine, but it has UNIX in it, here's what you can do:
Buy a new harddisk (it is cheap these days). Plug the new hdd into the borrowed machine and install Windows on it.
You can run all your test on it without disturbing its original UNIX !
At the end of the test, you can always install the hdd in your future new computer. Or you can buy an enclosure and use it as a portable back up disk.
Buy a new harddisk (it is cheap these days). Plug the new hdd into the borrowed machine and install Windows on it.
You can run all your test on it without disturbing its original UNIX !
At the end of the test, you can always install the hdd in your future new computer. Or you can buy an enclosure and use it as a portable back up disk.