Best machine for optimization (processor type, OS, others)?

Questions about MultiCharts and user contributed studies.
Tresor
Posts: 1104
Joined: 29 Mar 2008
Has thanked: 12 times
Been thanked: 51 times

Best machine for optimization (processor type, OS, others)?

Postby Tresor » 30 May 2008

Hello everyone,

I am just about to start a series of serious optimizations. My little laptop with Intel T7400 is too slow. Please see a print screen attached. It takes 61 hours to calculate one strategy on 3.5 year data (20 minute compression). I need to calculate at least 20 such strategies.

Can you advise me from your own experience what computer would be best for backtesting and optimization? I am not asking for an advice on a personal computer or a workstation - these will be too slow. I am asking for an advice on a server of high computing power.

I would appreciate any help on a specification of such a server (type of processors, number of processors (4/8/16/32?), operating system. If you were using servers for backtesting and optimization and you noticed a boost in performance over a personal computer, please let me know the name of this server. If you were using 2 servers and noticed that one overperformed the other, please tell me

Regards
Attachments
optimization.jpg
optimization.jpg (102.29 KiB) Viewed 373 times
Last edited by Tresor on 01 Jun 2008, edited 1 time in total.

User avatar
TJ
Posts: 6523
Joined: 29 Aug 2006
Location: Global Citizen
Has thanked: 957 times
Been thanked: 1883 times

Postby TJ » 31 May 2008

you can take a look at this test run:

http://www.elitetrader.com/vb/showthrea ... ost1849141

It was done on a Intel Q6600 quadcore processor.
You can see in the Task Manager that MultiCharts utilized all 4 CPUs to the max !

Tresor
Posts: 1104
Joined: 29 Mar 2008
Has thanked: 12 times
Been thanked: 51 times

Postby Tresor » 01 Jun 2008

you can take a look at this test run:

http://www.elitetrader.com/vb/showthrea ... ost1849141

It was done on a Intel Q6600 quadcore processor.
You can see in the Task Manager that MultiCharts utilized all 4 CPUs to the max !


Hello TJ,

Many thanks for the link. The machine Tums built is made of one processor. And from what I understand all four cores of this processor are 99% utilised while optimizing. Great news!

Do you think that when I build a machine of four Intel Q6600 processors the optimization that MC will be doing will be four times faster than the optimization run just on one Intel Q6600?

Regards

Tresor
Posts: 1104
Joined: 29 Mar 2008
Has thanked: 12 times
Been thanked: 51 times

Postby Tresor » 04 Jun 2008

I have one more question;

Is optimization faster when data is ASCII mapped or when data is ASCII imported?

Regards

User avatar
Marina Pashkova
Posts: 2758
Joined: 27 Jul 2007

Postby Marina Pashkova » 04 Jun 2008

I have one more question;

Is optimization faster when data is ASCII mapped or when data is ASCII imported?

Regards


Hi Tresor,

It doesn't matter where your data comes from. The source of the data doesn't affect the optimization speed at all.

Regards.

Tresor
Posts: 1104
Joined: 29 Mar 2008
Has thanked: 12 times
Been thanked: 51 times

Postby Tresor » 04 Jun 2008

Thanks Marina.

Anyone out there having experience with optimizing on a machine with more than one processor on board?

User avatar
Marina Pashkova
Posts: 2758
Joined: 27 Jul 2007

Postby Marina Pashkova » 09 Jun 2008

Hi Tresor,

Do you have any particular questions regarding optimization on multi-core machines?

Regards.

Tresor
Posts: 1104
Joined: 29 Mar 2008
Has thanked: 12 times
Been thanked: 51 times

Postby Tresor » 09 Jun 2008

Hi Tresor,

Do you have any particular questions regarding optimization on multi-core machines?

Regards.


Hi Marina,

There's is one question that I have had in my mind for a few days. Namely, I contacted Microsoft Support and a guy told me that even if I buy a machine with 4 CPUs, then XP Proffesional and Vista will only use maximum of 2 CPUs and will be blind for the remining 2 CPUs and will prevent any other application to utilise more than 2 CPUs.

In other words, I should buy a machine with no more than 2 CPUs. Can you confirm what this guy told me?

Regards

User avatar
Marina Pashkova
Posts: 2758
Joined: 27 Jul 2007

Postby Marina Pashkova » 09 Jun 2008

Hi Tresor,

According to this document http://www.microsoft.com/licensing/high ... icore.mspx what you're saying is true: more than 2 processors can't be used in XP and Vista.

To be honest, we're not sure if it really won't work. Actually, all 4 cores might be used in optimization on XP. I would recommend trying to run optimization on a quad-core machine and check the CPU load.

Or you could purchase Windows Server - it'll use all the available cores for optimization for sure.

Regards.

Tresor
Posts: 1104
Joined: 29 Mar 2008
Has thanked: 12 times
Been thanked: 51 times

Postby Tresor » 09 Jun 2008

Hi Tresor,

According to this document http://www.microsoft.com/licensing/high ... icore.mspx what you're saying is true: more than 2 processors can't be used in XP and Vista.

To be honest, we're not sure if it really won't work. Actually, all 4 cores might be used in optimization on XP. I would recommend trying to run optimization on a quad-core machine and check the CPU load.

Or you could purchase Windows Server - it'll use all the available cores for optimization for sure.

Regards.


Hi Marina,

That would make things different if I could use Windows Server as my operating system :D Having read this system requirements http://www.tssupport.com/multicharts/requirements/ I just thought that only operating systems that MC can work with are XP and Vista.

Regards

User avatar
Marina Pashkova
Posts: 2758
Joined: 27 Jul 2007

Postby Marina Pashkova » 10 Jun 2008

A follow-up to my previous post:

What Microsoft says on their website is true for multiple CPU's not multiple cores within a processor. In other words, if you have a single CPU with multiple cores, all of the cores will be used.

Regards.

Tresor
Posts: 1104
Joined: 29 Mar 2008
Has thanked: 12 times
Been thanked: 51 times

Postby Tresor » 10 Jun 2008

Hi all,

I have run 3 tests of VPCI strategy on on 3.5 year data, 16 minute time bars, with the same optimizable inputs. The results were as follows

1. Fujitsu-siemens laptop T4215 with 1 x Intel T7400, 2.16 GHz, 2GB RAM, XP 32 bit, time: 14:02:52

2. HP DL 140 server with 1 x Intel Xeon E5130, 2GB RAM, XP 64 bit, time: 10:37:05

3. Dell Power Edge 2950 server with 2 x Intel Xeon E5430, 2.55 GHz, 8GB RAM, Windows 2003 Server, time: 2:11:26

I attach the print screen of the last test. It is confirmed Portfolio Backtester can run on Windows Server, all 8 cores are 97% used :D

This test shows that the performance of Portfolio Backtester on 8 cores with Windows Server installed and 4 x higher RAM is 6.6 times quicker than on 2 cores of my laptop. Good news.

Now, did any of guys tried to optimize on a machine with more than 2 CPUs? If yes, then how the performance was increased?

Regards

BTW, Marina can you also add Windows Server operating system to System Requirements on your website. Maybe some of institutional customers or heavy optimizers might find this info useful.
Attachments
Dell 8 core optimization.jpg
Dell 8 core optimization.jpg (124.53 KiB) Viewed 364 times

User avatar
TJ
Posts: 6523
Joined: 29 Aug 2006
Location: Global Citizen
Has thanked: 957 times
Been thanked: 1883 times

Postby TJ » 10 Jun 2008

Tresor:

Thank you for sharing your test.

Good to see how 2x CPU produces better than 3x result.

FYI, on the last test, only 2.5GB of RAM was used. Therefore the additional RAM beyond 4GB was not a factor.

note: XP can utilize 3GB RAM. Most people install 4GB in their computers these days.

note: if the program requires 2.5GB, and your computer must swap memory because it only has 2GB, then the penalty in swapping is very high.
I would be interested to see how your second test would perform, if the computer has 4GB of memory. I think it will take less than half the time it took.


best regards
TJ

Tresor
Posts: 1104
Joined: 29 Mar 2008
Has thanked: 12 times
Been thanked: 51 times

Postby Tresor » 10 Jun 2008

Tresor:

Thank you for sharing your test.

Good to see how 2x CPU produces better than 3x result.

FYI, on the last test, only 2.5GB of RAM was used. Therefore the additional RAM beyond 4GB was not a factor.

note: XP can utilize 3GB RAM. Most people install 4GB in their computers these days.

note: if the program requires 2.5GB, and your computer must swap memory because it only has 2GB, then the penalty in swapping is very high.
I would be interested to see how your second test would perform, if the computer has 4GB of memory. I think it will take less than half the time it took.


best regards
TJ


Hi TJ,

It really seems that buying a computer with more than 4 GB RAM would be throwing away the money as Backtester uses 2.5 GB. Thanks for this advice.

I already am searching for a machine in my vicinity that has 4 (or more) CPUs x 4 cores that I can install MC on and put to test. The ctach is here though that such machines big have Unix installed, not Windows. So my search will be a rather lenghthy one.

As soons as I get my hands on such a machine I will post the results.

Regards

User avatar
TJ
Posts: 6523
Joined: 29 Aug 2006
Location: Global Citizen
Has thanked: 957 times
Been thanked: 1883 times

Postby TJ » 10 Jun 2008

if you can "borrow" a powerful Intel machine, but it has UNIX in it, here's what you can do:

Buy a new harddisk (it is cheap these days). Plug the new hdd into the borrowed machine and install Windows on it.

You can run all your test on it without disturbing its original UNIX !

At the end of the test, you can always install the hdd in your future new computer. Or you can buy an enclosure and use it as a portable back up disk.

Tresor
Posts: 1104
Joined: 29 Mar 2008
Has thanked: 12 times
Been thanked: 51 times

Postby Tresor » 10 Jun 2008

Thanks TJ

Will do this :)


Return to “MultiCharts”