2 Computers - How to speed up performance in Multicharts?

Questions about MultiCharts .NET and user contributed studies.
whirl
Posts: 22
Joined: 16 Aug 2010
Has thanked: 4 times

2 Computers - How to speed up performance in Multicharts?

Postby whirl » 23 Dec 2015

Hello Everyone. I have 2 different computers running Portfolio Trader using Multicharts .NET trading 100 sp100 stock symbols. I am only using End of day data to generate trades. I changed the time the exchange closes to a few minutes before the market actually closes, so when it does generate trades, it can send them before the real stock market closes.

Now, I am running the same identical strategy, same list of symbols, same data provider (iqfeed), same broker (using ib gateway), configured Multicharts (same versions) the same exact way. So basically everything about Multicharts and its configurations are the same for both computers.
However, on the older computer, on Portfolio Trader, when I click Automate Order Execution, it literally takes less than 10 seconds (often 5) and it's enabled.
However on the newer computer, it takes anywhere from 30 seconds to 10 minutes to turn on! I am trying to figure out why and what it takes to make the new computer as fast as the old computer. Everything else about the new computer seems up to speed, I don't see a problem, however sometimes trades on the new computer do not go thru like they do on the old computer, especially after a few days. It's driving me crazy, as I can't figure out why the newer computer is 5x slower than the old one.

Ok here are the specs of the old computer:
Intel I7 2600k @3.4 ghz (quad core)
16 gigs Ram
Windows 7 Professional 64 bit
1 TB 7200 hard drive (conventional)
In addition to running MC it has some random stuff on it, browser, aim, ToS software etc.
But it runs MC very quickly.

New computer:
Intel Xeon e5-2650 @2GHZ (8 core)
32 gig RAM
Windows 10 (freshly installed with all the latest updates)
500gig hd (conventional 7200 rpm)
The only thing installed on this fresh windows 10, is MC.net, IB Gateway, and IQ feed. No other software.
I disabled as many bells and whistles as I could.
It seems to run fine on typical tasks.
However, when I initiate Automated Trading in Portfolio Trader - I notice the hard drive spin a bit more, and it takes a little longer just to get downloading the quote data. Once downloaded, it then takes a few seconds more (kind of like hour glass mode), while it does the calculations, then FINALLY auotrading is enabled. So whereas the old computer took about 7 seconds to load, the new one takes about 28 seconds.

Now, by itself 28 seconds to load the autotrade isn't that big of a deal. However, when the closing bell occurs, ALL of my entry and exit trades are generated so it could be like 10 trades happening at the same second, so all the calculations for 100 symbols and its indicators have to happen the exact moment the closing bell occurs (remember I set this for a few minutes before the market actually closes). On the old computer, there never seems to be a problem keeping up w/ all the calculations and theoretically it can run for a few weeks without a hiccup.
However, on the new computer, it seems to be a lot slower 5 to 10x slower. So where the old computer can handle all those trades happening at once, the newer computer can not handle all these calculations I guess because it is running slower, and a % of my trades never get generated and sent to the broker.

This newer computer, should be a lot faster than the 4 year older one., it has more cores, more ram, a cleaner operating system w/ no junk software on it. While the old computer has so much random software on it, and still runs circles around the new one in terms of Multicharts at least.
When I run something like Windows Experience Index, the new one has scores which are faster in all areas. So I have NO idea why this new computer is lagging, and the hard drive spins more often as it's doing its tasks. I don't have any special firewall/antivirus other than the default defender and firewall that comes w/ windows 10.

Does anybody have any idea what I can do or configure in the new computer to speed it up please?
I'm not sure what else I can do, all the bios and drivers are up to date.
thank you very much

J

whirl
Posts: 22
Joined: 16 Aug 2010
Has thanked: 4 times

Re: 2 Computers - How to speed up performance in Multicharts

Postby whirl » 23 Dec 2015

Here's an update. I installed MC.net on a different i7. And it also ran exceptionally fast, like 4 seconds vs 30 seconds to enable auto trading. Could it be that the Xeon processors weren't designed to run something like Multicharts quickly? It works fine for backtesting and optimisation w/ all of the extra cores, but not when running the live trading? Does anybody have experience with comparing Xeon w/ I7s?

thank you

shane1800
Posts: 80
Joined: 28 Feb 2014
Has thanked: 5 times
Been thanked: 5 times

Re: 2 Computers - How to speed up performance in Multicharts

Postby shane1800 » 06 Feb 2016

Any new learnings from your testing?

Was the Xeon faster than i7 at genetic optimizing?

Wondering about a dual cpu Xeon board for testing.

whirl
Posts: 22
Joined: 16 Aug 2010
Has thanked: 4 times

Re: 2 Computers - How to speed up performance in Multicharts

Postby whirl » 07 Feb 2016

For optimising, the Xeon with more cores was faster for sure.
For live testing, I decided to go w/ the I7.

So that's how I ended up using it.

The9000
Posts: 59
Joined: 24 Oct 2015
Has thanked: 2 times
Been thanked: 4 times

Re: 2 Computers - How to speed up performance in Multicharts

Postby The9000 » 20 Feb 2016

I'm running MC 9.1 in an EC2 c4 compute instance which is Xeon 2666 chips on Windows Server 2012.

Normally for compute intensive backtests, I scale up from a c4.large (2 cpus, 4GB ram) to a c4.2xlarge (8 cpus, 16gb ram) and I have similar problems when I go to the c4.2xlarge. It actually runs backtests slower with extra CPU horsepower on the larger instances and there is a substantial lag to load the data and begin the actual backtest. This is with the Portfolio Trader using the PowerLanguage version of MC.

On TS 9.5, going from c4.large to c4.2xlarge substantially speeds up backtests.

whirl
Posts: 22
Joined: 16 Aug 2010
Has thanked: 4 times

Re: 2 Computers - How to speed up performance in Multicharts

Postby whirl » 20 Feb 2016

That's interesting. For backtests and optimisation, the extra Xeon cores seem to help me. Although I am using the .Net version.
It's only when when I enable the autotrading does the Xeon chip seem to run slower.
If you enable autotrade (even if put a tiny amount using live real money as a test), does it run faster or slower than backtesting?

shane1800
Posts: 80
Joined: 28 Feb 2014
Has thanked: 5 times
Been thanked: 5 times

Re: 2 Computers - How to speed up performance in Multicharts

Postby shane1800 » 20 Feb 2016

I've had a difficult time with vps instances also. Wasn't sure why it got slower with larger instances so decided on an overclocked 5960x which works great.

whirl
Posts: 22
Joined: 16 Aug 2010
Has thanked: 4 times

Re: 2 Computers - How to speed up performance in Multicharts

Postby whirl » 20 Feb 2016

a lot of those vps places, were designed for things like database references or webservers where access was more important than processing power. dont think those would be useful for backtesting large amounts of data, so yea i think finding your own processor for your own needs is probably the route to go

The9000
Posts: 59
Joined: 24 Oct 2015
Has thanked: 2 times
Been thanked: 4 times

Re: 2 Computers - How to speed up performance in Multicharts

Postby The9000 » 20 Feb 2016

I use the c4.large for live trading and it doesn't have any delay.

The c4 AWS instances were designed specifically for compute intensive functions. The c4 instances are substantially faster than the c3 instances for about the same price. The downfall of both of them is they need more ram.

The Xeon 2666 chip in the c4 instances is bad ass.

shane1800
Posts: 80
Joined: 28 Feb 2014
Has thanked: 5 times
Been thanked: 5 times

Re: 2 Computers - How to speed up performance in Multicharts

Postby shane1800 » 21 Feb 2016

I made a benchmark workspace to test our computers optimization speed using exhaustive.


After a few minutes I got 79 minutes as expected finish time using an overclocked 5960x at 46x.

You?
Attachments
!Benchmark-MC.wsp
(202.17 KiB) Downloaded 416 times

The9000
Posts: 59
Joined: 24 Oct 2015
Has thanked: 2 times
Been thanked: 4 times

Re: 2 Computers - How to speed up performance in Multicharts

Postby The9000 » 22 Feb 2016

On a c4.large AWS instance it took 3:40 (3 minutes 40 seconds) to hit the 5000 tests mark. I will try it on a c4.2xlarge instance the next time I'm using one.

The9000
Posts: 59
Joined: 24 Oct 2015
Has thanked: 2 times
Been thanked: 4 times

Re: 2 Computers - How to speed up performance in Multicharts

Postby The9000 » 26 Feb 2016

On a c4.2xlarge instance, it took 56 seconds to get to the 5000 mark and the estimate was 197:00:00 to completion.

Shane can you run the script and see how long it takes to get to the 5000 mark? If your 5960x is really that fast, I'll be on my way to Microcenter to build a bad ass R&D rig.

shane1800
Posts: 80
Joined: 28 Feb 2014
Has thanked: 5 times
Been thanked: 5 times

Re: 2 Computers - How to speed up performance in Multicharts

Postby shane1800 » 26 Feb 2016

Cool numbers.

My computer will be free to run that test again in about a week, will let you know.

I got about a 19,000 on CPU Mark overclocked to 4500 mhz.
https://www.cpubenchmark.net/high_end_cpus.html

The newest fastest chip, i7-6950x should be coming out in a few months.

I may get a dual Xeon board to test one day.

The9000
Posts: 59
Joined: 24 Oct 2015
Has thanked: 2 times
Been thanked: 4 times

Re: 2 Computers - How to speed up performance in Multicharts

Postby The9000 » 26 Feb 2016

The interesting thing is the c4.large instances use dual Xeon E5 2666 chips, which in a dual CPU config has a benchmark of 24804 @ 2.9 Ghz.

Point being, for the purpose of Multicharts, the 5960x runs circles around an army of 8 E5 2666 chips.

shane1800
Posts: 80
Joined: 28 Feb 2014
Has thanked: 5 times
Been thanked: 5 times

Re: 2 Computers - How to speed up performance in Multicharts

Postby shane1800 » 26 Feb 2016

I've even tried optimizing on a 64 core server at profitbricks and it didn't beat the 5960x. Guessing there's something with Xeon's or with virtualization...

Same results using TS.

shane1800
Posts: 80
Joined: 28 Feb 2014
Has thanked: 5 times
Been thanked: 5 times

Re: 2 Computers - How to speed up performance in Multicharts

Postby shane1800 » 02 Mar 2016

Took about 25 seconds to do 5,000. 79 hours to completion.

I see that Ec2 offers few faster C4 instances so maybe they can match a 5960x.

For overclocking note that all chips are different so a good way to guarantee performance is to buy from a place like this.

https://siliconlottery.com/products/5960x45g

Test results from Passmark and Intel test.
https://www.cpubenchmark.net/high_end_cpus.html
Performance test 19390

Intel test 2313
Attachments
Screen Shot 2016-03-02 at 11.19.45 AM.png
(77.47 KiB) Downloaded 5906 times

The9000
Posts: 59
Joined: 24 Oct 2015
Has thanked: 2 times
Been thanked: 4 times

Re: 2 Computers - How to speed up performance in Multicharts

Postby The9000 » 21 Apr 2016

So I followed suit and put together an overlocked 5960x rig for my R&D machine. Asus Sabertooth MB, 32gb ram with enough cooling to keep a small nuclear power plant stable. It's running stable at 4.3 Ghz and passed a stress test without breaking a bead of sweat.

I did a showcase showdown against the C4 instances on AWS and the 5960x is comparable, performance wise, to a 16 vCPU c4 instance. What's interesting is when I'm running a full backtest on this machine that takes 25 minutes, the CPU is only at 8-9% utilization. It seems like there is a lot more meat on the bone if there is a way to utilize more cores on this beast.

shane1800
Posts: 80
Joined: 28 Feb 2014
Has thanked: 5 times
Been thanked: 5 times

Re: 2 Computers - How to speed up performance in Multicharts

Postby shane1800 » 21 Apr 2016

That is interested you're not getting 100% on the 5960x. Are you using MC 64bit?
Attachments
Screen Shot 2016-04-21 at 8.00.59 AM.png
(139.04 KiB) Downloaded 5880 times

The9000
Posts: 59
Joined: 24 Oct 2015
Has thanked: 2 times
Been thanked: 4 times

Re: 2 Computers - How to speed up performance in Multicharts

Postby The9000 » 21 Apr 2016

I'm using MC64 Portfolio Trader v9.1 release 4, Windows 10 Pro x64. I used the Intel Extreme Tuning Tool to watch the CPU/memory load while it was running, I'll try task manager so we're comparing apples to apples.

The algo I ran it on was 30 symbols, 2 minute bars, 700 lines of code plus a MMS script. There is plenty going on to not insult the 5960x of it's capabilities...

My EC2 instance (c4.large) for live trading is on MC64 Portfolio Trader 9.1 release 3.

When running an optimization/backtest on the EC2 instance, according to the EC2 dashboard, it usually maxes out around 60% CPU load. I would guess that's because using all of the virtual CPUs would require code specific to multiple CPUs (such as virtualization, etc.).

shane1800
Posts: 80
Joined: 28 Feb 2014
Has thanked: 5 times
Been thanked: 5 times

Re: 2 Computers - How to speed up performance in Multicharts

Postby shane1800 » 21 Apr 2016

I faintly remember having the same issue before, MC not using 100% of the processor. It fixed itself but not sure how.

Maybe PT needs to be opened as Admin.

I remember my overclocking settings were looking good but MC would not use the processor fully, I overclocked it differently somehow and it started to work.

I'm using the latest version of MC PT.

hughesfleming
Posts: 238
Joined: 22 Apr 2014
Has thanked: 58 times
Been thanked: 61 times

Re: 2 Computers - How to speed up performance in Multicharts

Postby hughesfleming » 21 Apr 2016

Interesting discussion. I was under the impression that Portfolio trader was not multi threaded and would not benefit from many cores. Has something changed?

Are there any core limits when dealing with dual cpu systems?

Thanks,

Alex

shane1800
Posts: 80
Joined: 28 Feb 2014
Has thanked: 5 times
Been thanked: 5 times

Re: 2 Computers - How to speed up performance in Multicharts

Postby shane1800 » 22 Apr 2016

I'm curious about dual xeon CPU's also.

PT has worked for me using all my cores in the last year I've been using MC.

The9000
Posts: 59
Joined: 24 Oct 2015
Has thanked: 2 times
Been thanked: 4 times

Re: 2 Computers - How to speed up performance in Multicharts

Postby The9000 » 22 Apr 2016

I confirmed it's running at 8% CPU load.
Attachments
Screen Shot 2016-04-22 at 8.36.46 AM.png
(406.51 KiB) Downloaded 5894 times

shane1800
Posts: 80
Joined: 28 Feb 2014
Has thanked: 5 times
Been thanked: 5 times

Re: 2 Computers - How to speed up performance in Multicharts

Postby shane1800 » 22 Apr 2016

Maybe it's because the of the type of optimization you're running? Try the benchmark test to see if you hit 100%.

My portfolio tree looks like this, maybe each is on a core?
Attachments
Screen Shot 2016-04-22 at 8.14.16 AM.png
(33.41 KiB) Downloaded 5901 times

The9000
Posts: 59
Joined: 24 Oct 2015
Has thanked: 2 times
Been thanked: 4 times

Re: 2 Computers - How to speed up performance in Multicharts

Postby The9000 » 22 Apr 2016

I ran the benchmark algo in regular MC64x and it does peg at 100%.

Test results were 25 seconds to the 5000 mark and 83 hours to estimated completion. It still smokes the AWS instance by a country mile.

I'm going to try breaking up my test algo that I'm running on the entire S&P400 MidCap into 16 chunks, to see if that helps.

User avatar
ABC
Posts: 636
Joined: 16 Dec 2006
Has thanked: 120 times
Been thanked: 359 times
Contact:

Re: 2 Computers - How to speed up performance in Multicharts

Postby ABC » 24 Apr 2016

The9000,

do you use the words backtest and optimization to describe the same thing? Because if not and you actually did a backtest (and not an optimization) in your test where the CPU reached 8% utilization (was this overall or for one core?), this is not too surprising as a backtest has to be done sequential and can't be split up across cores. In an optimization you can run one combination per core (which in itself would still be sequential) and therefore utilize each core.

Regards,

ABC

User avatar
Henry MultiСharts
Posts: 9136
Joined: 25 Aug 2011
Has thanked: 1260 times
Been thanked: 2929 times

Re: 2 Computers - How to speed up performance in Multicharts

Postby Henry MultiСharts » 25 Apr 2016

Dear users,

Please study the following post:
viewtopic.php?f=16&t=10811#p55824

In order to utilize all cores for optimization you need to specify the number of cores you have in
decimal system for the "NumberOfThreadsOnOptimization" key in
HKEY_CURRENT_USER\Software\TS Support\Power Language\StudyRunner

Whether your CPU will be loaded to the maximum or not depends on your signal and optimization setup.

shane1800
Posts: 80
Joined: 28 Feb 2014
Has thanked: 5 times
Been thanked: 5 times

Re: 2 Computers - How to speed up performance in Multicharts

Postby shane1800 » 25 Aug 2016

Hi,

I'm trying to get MC to use 2 Xeon CPU's but on optimization and backtesting only 50% of the CPU total is used.

I tried importing this regedit and using the all threads for optimization dword:00000000 and also 24 in decimal as there are 24 threads for this server. Same results even after rebooting and running MC as admin.

viewtopic.php?f=19&t=45810&p=100648&hil ... on#p100648

I have used another software which runs at 100% and all 24 threads so the server should be fine.
Attachments
Screen Shot 2016-08-25 at 11.32.15 AM.png
(72.49 KiB) Downloaded 5811 times

The9000
Posts: 59
Joined: 24 Oct 2015
Has thanked: 2 times
Been thanked: 4 times

Re: 2 Computers - How to speed up performance in Multicharts

Postby The9000 » 25 Aug 2016

Keep us posted please if you can get it to work.

I'm also curious to see if you can use all of the threads. These second hand dual Xeon setups are getting too cheap to not take advantage of for brute force optimizations or backtests that require a metric crap ton of RAM.

http://www.natex.us/Intel-S2600CP2J-Dua ... -12800.htm

For $499 with 128gb of ram, it's a bargain. Especially with 16 cores and 32 threads. With 10 core Xeons that are less power hungry (20 core, 40 thread total), the same setup is around $1k. Still a bargain.

The single core performance on these isn't anything to get excited about (backtesting) and can't hold court with an overclocked 5960x, however for brute force optimizations, it's the best bang for the buck.

shane1800
Posts: 80
Joined: 28 Feb 2014
Has thanked: 5 times
Been thanked: 5 times

Re: 2 Computers - How to speed up performance in Multicharts

Postby shane1800 » 25 Aug 2016

Yes will keep everyone updated!

I did notice that all the cores are running, 12 shown in the attachment, but just at 50%. So it looks like MC is using both CPU's but just not fully. Maybe hyper threading is not working for MC or maybe MC is not using all the ram available and creating a bottle neck?

All ram slots on the board are being used though.
Attachments
Screen Shot 2016-08-25 at 11.31.45 AM.png
(27.79 KiB) Downloaded 5813 times

The9000
Posts: 59
Joined: 24 Oct 2015
Has thanked: 2 times
Been thanked: 4 times

Re: 2 Computers - How to speed up performance in Multicharts

Postby The9000 » 25 Aug 2016

I've noticed it also depends on the type number of symbols you're trying to optimize on and the strategy. Depending on the type of optimization I'm doing, the 8 core 5960x will run at between 30% - ~95%. It also takes awhile for all of the cores to become active in the optimization. It might run at 30% for the first few minutes and pickup steam as it goes on.

When all 8 cores are active, it's a ruthlessly efficient machine. When it's doing a single core back test (usually 8% CPU load), it doesn't even go into turbo and rarely the clock speed gets over 2.8 Ghz. I have it overclocked at 4.2 Ghz. When running optimizations, it runs at 4.2 Ghz.

Especially if I'm starting with a large chunk of data (8GB+). Such as the entire SPY500 in 2 minute bars for 2-3 years.

shane1800
Posts: 80
Joined: 28 Feb 2014
Has thanked: 5 times
Been thanked: 5 times

Re: 2 Computers - How to speed up performance in Multicharts

Postby shane1800 » 25 Aug 2016

I appreciate that advice as it's working now!

I am running a PT genetic opt using 4 different strategies and now all cores are 100%. So that was the issue, using a simple optimization did not use all cores. Not sure if changing the regedit code did anything.

I'm looking forward to getting a new server :)
Attachments
Screen Shot 2016-08-25 at 6.11.45 PM.png
(89.22 KiB) Downloaded 5817 times

eunos64
Posts: 19
Joined: 15 Sep 2013
Has thanked: 10 times
Been thanked: 1 time

Re: 2 Computers - How to speed up performance in Multicharts

Postby eunos64 » 22 Oct 2016

Hello,
I have an opportunity to try testing 2 Xeon = 24cores 48threads CPU machine.
The result is fine, CPUs works at 80 to 100% while performing strategy optimization.
But, I think, I need further tweak on memory usages. Because it seems that larger data MC setted needs larger memory size.

Testing environment is below...
My PC: Single Xeon 8 cores 16 threads, 32G memory, win10 64 pro, MC64.net v9.1 r4,
Test PC: Dual Xeon 12 cores 24 threads (24C/48T), 32G memory, same above... maybe 3 to 4 times faster calculation ability than My PC.
Performing Strategy optimization with 3032 Tokyo Stock Exchange symbols , Daily OHLC, 1000 bars back, 1250 strategy input.

Status of PC usage:
My PC: http://www.multicharts.com/discussion/d ... ew&id=9978
Test PC unfortunately CANNOT finish optimization. Test gradually exsausts 32/32GB memory and Windows10 said "ERROR, short of memory" , then PC screen has blacked out ... could not operate anymore...

So, I reduced data 3032 symbols to 472 symbols. It works fine. see below,
http://www.multicharts.com/discussion/d ... ew&id=9977

MC team
I want to know it can be completed if I installed more memory.

Best regards, thanks.
Attachments
Test_optimization_3032symbols_1000bars_1250.jpg
(253.98 KiB) Downloaded 5795 times
Test_optimization_472symbol_1000bars_1250.jpg
(681.01 KiB) Downloaded 5793 times

The9000
Posts: 59
Joined: 24 Oct 2015
Has thanked: 2 times
Been thanked: 4 times

Re: 2 Computers - How to speed up performance in Multicharts

Postby The9000 » 22 Oct 2016

If you're asking if you can use more than 32gb, yes. With so many cores, you're likely choking that machine with the memory.

I have a strategy that uses a metric ton of ram and I was running into crashes when it starts paging at 64gb. My 5960x rig only supports 64gb of ram and I needed something more. So I put one of these together:

http://www.natex.us/Intel-S2600CP-Mothe ... -128gb.htm

The $475 price tag for the e5-2670 CPUs, MB and 128gb of ECC ram was too tempting. I picked up the CPU coolers and power supply from Natex too.

All in, I have about $900 in the rig with case, power supply, SSD and a few other odds and ends to make this no frills server board work in a civilian case. Not bad for 16 cores and 128gb of ram, considering I spent 3x that on the 5960x rig with 64gb of DDR4.

I ran the above benchmark on it and it clocks in at 29 seconds, only a few seconds behind the 5960x @ 4.3 Ghz. However in a backtest, the 5960 out paces it by ~35% because of the much higher single core throughput.

I'm very impressed for the price.

wilkinsw
Posts: 558
Joined: 21 Apr 2013
Has thanked: 139 times
Been thanked: 91 times

Re: 2 Computers - How to speed up performance in Multicharts

Postby wilkinsw » 04 Nov 2016

Re Speed.........

1) Very affordable 4c/8thread Haswell i7s should do the provided benchmark optmisation in about 40 seconds... So I'd recommend to just load up on those.

2) If squeezing performance out of a server, make sure to turn on the "high performance" power plan! Also to prioritise programs and not background processes (sysdm.cpl .... advanded etc etc). Doing these two things will almost double the optmisation speed of your server!


Return to “MultiCharts .NET”