Real life optimization speed

Questions about MultiCharts and user contributed studies.
glam_100
Posts: 157
Joined: 14 Jun 2006
Has thanked: 2 times
Been thanked: 9 times

Real life optimization speed

Postby glam_100 » 14 Jul 2007

Hi,

I'm a long time TS user and I'm thinking of migrating to MC because of the exciting multi-core optimization feature.

For those who have done optimizations on MC, may I ask what's the real life optimization speed you're getting compared to TS? I'm interested in both the multi-core speed and single core speed as compared to TS optimization. The single core speed would tell me whether MC optimization is more efficient than TS on a single core.

I'm very happy with TS except for the optimization speed. I have no problem with the lack of portfolio back-test or lack of genetic optimization cos I have bought RINA and other 3rd party genetic optimizers. What really gets me is the ridiculous single core usage in TS optimization. I'm a very heavy user of optimization and anything that could potentially give me 4x the speed (on 4 cores machine) is definitely worth looking at...

glam_100
Posts: 157
Joined: 14 Jun 2006
Has thanked: 2 times
Been thanked: 9 times

Postby glam_100 » 14 Jul 2007

I just downloaded MC beta and played around with it.

I loaded up a few signals and optimize to test for speed. The first time the optimization went ok, but from the 2nd time on the optimizer just quit on me. It jumps out from the optimization window. First it jumps out after a few thousands iterations, then it jumps out sooner and to the point of jumping out of optimiztion immediately after I click to optimize 'ok' button.

Obviously there is a bug with MC optimization.

While my first impression with MC is great and looks like it picks up from where TS has left this piece of software has some serious bug issues to solve before it can be used for research in serious day to day operation let alone to be used as an ATS environment...

glam_100
Posts: 157
Joined: 14 Jun 2006
Has thanked: 2 times
Been thanked: 9 times

Postby glam_100 » 15 Jul 2007

I played around with a few optimizations (keltner channel, bollingerband) and compared the speed between TS and MC...

What I found is basically the speed of TS using 1 core is the same as MC using 2 cores...

This is quite surprising because it suggest EL code compiled into C++ on MC is only half efficient as TS implementation...

With a 2 core machine I get no advantage in optimization speed...

Tssupport, could you see if there is anything you can do with that? I'm counting on MC to give me multi core optimization that TS could'n't give us for over 5 years...

But having code run half as efficient as TS is a great penalty - I get no improvement with 2 cores, and with 4 cores I only get <2x improvement after all the migration trouble

glam_100
Posts: 157
Joined: 14 Jun 2006
Has thanked: 2 times
Been thanked: 9 times

Postby glam_100 » 15 Jul 2007

BUG report

I found that everytime I cancel an optimization, MC runs into optimization premature exit for the next run.

TSsupport pls look into the problem...

scenario reproduce:

1) run an exhaustive optimization and cancel it
2) run another optimization under genetic mode

what to expect:

In the second run of optimization (genetic mode), either the optimizer exit prematurely or it doesn't show optimization report

User avatar
Andrew Kirillov
Posts: 1589
Joined: 28 Jul 2005
Has thanked: 2 times
Been thanked: 31 times
Contact:

Postby Andrew Kirillov » 16 Jul 2007

glam_100,
The problems you encounter make me thinks that you use wrong version.
What version of the product are you running?
The latest beta works faster than TS8.3 both on a single core or a dual core.
Please confirm your version and we will proceed.
Last edited by Andrew Kirillov on 19 Jul 2007, edited 1 time in total.

glam_100
Posts: 157
Joined: 14 Jun 2006
Has thanked: 2 times
Been thanked: 9 times

Postby glam_100 » 16 Jul 2007

Hi Andrew,

The version I'm using is this:

2.1.880.530

glam_100
Posts: 157
Joined: 14 Jun 2006
Has thanked: 2 times
Been thanked: 9 times

Postby glam_100 » 16 Jul 2007

Just found out the latest version is 2.1.909.909 ...

Gee... you guys are really rolling things out fast... I think I downloaded just 2 weeks ago...

I will try the latest version and report back

glam_100
Posts: 157
Joined: 14 Jun 2006
Has thanked: 2 times
Been thanked: 9 times

Postby glam_100 » 16 Jul 2007

Hi Andrew,

I just downloaded the latest Beta and performed the following test:

Symbol: SPY
Date: 7/18/2005 - 7/17/2007 (2 years back)
Strategy and Optimization parameters:
MovAvg 2 Line Cross LE ( FastLength: 1..9; Slow Length: 1..18 )
MovAvg 2 Line Cross SE ( FastLength: 1..9: Slow Length: 1..18 )
Total Number of tests: 26,244
Hardware: Core 2 Duo e6600 @ 3.7Ghz with 2GB memory

Result:

TS 8.2 Build 3896 - 1 min 19 seconds
Multi Chart 2.1.909.909 - 1 min 21 seconds

So MC running on 2 cores is roughly the speed of TS running 1 core.

Could you please take a look at what goes wrong? I am really excited to learn that MC is suuposed to run faster than TS core to core...

If MC is working as you suggest with faster per core speed than TS and support multi-core I'm sure many many many TS users will jump right at MC in no time... it has been a big issue for many TS users for at least 4 years and TS has not made any response at all!

glam_100
Posts: 157
Joined: 14 Jun 2006
Has thanked: 2 times
Been thanked: 9 times

Postby glam_100 » 16 Jul 2007

Hi Andrew,

I tried another optimization and encountered 2 bugs:

The first bug I tried to run the following optimization:

BollingerBand LE ( Length: 1..50; Num StdD: 1..5)
BollingerBand SE( Length: 1..50; Num StdD: 1..5)

Total tests: 62500

When I run this optimization MC hangs up every time when it just gets past 30000 tests... I restared computer and retry and it's the same.

Could you please take a look?

The second bug I tried to run another optimization with different parameters after the Mov Avg 2 Line Cross optimization mentioned above. MC just jump to the optimization report page without doing the actual optimization.

Could you also please take a look.

I really hope the MC optimization functions will work perfectly with no problem. Having stable bug free and multi core fast optimization will make MC a much more powerful research tool over TS... not to mention the integrated genetic optimizer...

Very good job indeed TS Support...

glam_100
Posts: 157
Joined: 14 Jun 2006
Has thanked: 2 times
Been thanked: 9 times

Postby glam_100 » 16 Jul 2007

Hi Andrew,

I did further testing. Both bugs seem to relate to the same source. Everytime I run an optimization with # simulation > 20000 - 30000 runs the above two problems appear.

You can 100% recreate the scenarios by running a BollingerBand LE and BollingerBand SE strategies with optimization parameter Length 1..50 and Std Dev 1..5 totaling 62500 tests...

User avatar
Andrew Kirillov
Posts: 1589
Joined: 28 Jul 2005
Has thanked: 2 times
Been thanked: 31 times
Contact:

Postby Andrew Kirillov » 17 Jul 2007

glam_100,
We are analyzing the problems you reported.

Speed problem is confirmed. We are analyzing what caused this problem since it is a fact the speed was higher in the internal builds of the program.
I will keep you posted.

glam_100
Posts: 157
Joined: 14 Jun 2006
Has thanked: 2 times
Been thanked: 9 times

Postby glam_100 » 17 Jul 2007

Hi Andrew,

Thanks for the reply. I'm looking forward to using Multi Chart for my research

User avatar
Andrew Kirillov
Posts: 1589
Joined: 28 Jul 2005
Has thanked: 2 times
Been thanked: 31 times
Contact:

Postby Andrew Kirillov » 17 Jul 2007

Glam_100,
Our research is coming to its completion so I can tell you some things...
First of all, we've found that there is a bug (a memory leak) in 909 version. It caused two problems - slow-down and the weird behavior you and O66 explained. I appreciate your help.

Also, it is a fact that the next MC beta version will be quicker than TS8 on dual core PC for all studies. However, it is important to understand that speed improvement varies: Bollinger bands signals work two times faster whereas Moving Averages will work just a little quicker on dual cores since we calculate it slower than TS8 does on a single core.


There is more room for improvement we can see and it is planned for MC 2.2

In a few days we will release a new beta build that doesn't contain the problems you encountered.

glam_100
Posts: 157
Joined: 14 Jun 2006
Has thanked: 2 times
Been thanked: 9 times

Postby glam_100 » 17 Jul 2007

Hi Andrew,

Thanks for your update.

It is exciting to hear a faster version is coming out soon and there is even more room for improvement in version 2.2

I will test it out once it's available

Thanks!

User avatar
Andrew Kirillov
Posts: 1589
Joined: 28 Jul 2005
Has thanked: 2 times
Been thanked: 31 times
Contact:

Postby Andrew Kirillov » 18 Jul 2007

Please keep in mind that we are releasing MultiCharts 2.1. MultiCharts 2.2 is planned for October-November.

glam_100
Posts: 157
Joined: 14 Jun 2006
Has thanked: 2 times
Been thanked: 9 times

Postby glam_100 » 18 Jul 2007

I just downloaded the new beta. The bugs are gone but the optimization speed is still a disappointment.

I ran a few benchmarks:

1) Moving Avg 2 Lines

Moving Avg 2 Line Cross LE ( Fast Len 1..9; Slow Len: 1..18 )
Moving Avg 2 Line Cross LE ( Fast Len 1..9; Slow Len: 1..18 )

TS time (with 1 core): 1 min 21 Sec
MC time (with 2 cores): 1 min 40 Sec

2) Bollinger Band

Bollinger Bands LE ( Len: 1..30; StdDev: 1..5 )
Bollinger Bands SE ( Len: 1..30; StdDev: 1..5)

TS time (with 1 core): 1 min 19 sec
MC time (with 2 cores): 46 sec

3) Price Channel with ATR Trailing

Price Channel LE ( Len: 1..2 )
Price Channel SE ( Len: 1..2 )
ATR Trailing LX ( Len: 1..20; NumATR: 1..3 )
ATR Trailing SX ( Len: 1..20; NumATR: 1..3 )

TS time (with 1 core): 21 secs
MC time (with 2 cores): 1 min 4 secs

4) MACD

MACD LE ( Fast: 1..12, Slow: 26; Len: 1..9 )
MACD LE ( Fast: 1..12, Slow: 26; Len: 1..9 )

TS time (with 1 core): 33 secs
MC time (with 2 core): 51 secs

It seems to me that most functions on MC is slower or significantly slower except for bollinger bands. More importantly, TS is running with just 1 core and MC is running with 2 cores.

While MC has given and still gives me dreams about faster optimization speed, I have to say the current optimization performance is a lot slower than TS. The most striking result is the third test where TS outperform MC 3 times with a single core while MC is using both cores. That would suggest MC is 6x less efficient on a core to core basis (at least for Prince channel and ATR trailing functions).

That shatters my dream of benefiting from faster optimzation in the near future. It looks like I'm stuck with TS, at least for the moment...

MC would make a very valuable research tool if its optimization speed can get at least on par with TS on a core to core basis due to MC's support on multi-core... but right now it just doesn't do it

User avatar
Andrew Kirillov
Posts: 1589
Joined: 28 Jul 2005
Has thanked: 2 times
Been thanked: 31 times
Contact:

Postby Andrew Kirillov » 19 Jul 2007

Glam_100,
Thanks for the analysis.
I will do me research, consult to engineers and get back to you shortly.

User avatar
Andrew Kirillov
Posts: 1589
Joined: 28 Jul 2005
Has thanked: 2 times
Been thanked: 31 times
Contact:

Postby Andrew Kirillov » 19 Jul 2007

Unfortunately I must admit that my conclusions grounded on wrong assumption. MultiCharts calculates indicators quicker than TS8.3 does and I trusted that it is true for signals as well. However signal engine is totally different and includes calculation of performance ratios such as net profit and similar. We have significant inefficiencies here and will optimize it in MC 2.2.

So I confirm that your tests reflect performance of MultiCharts and TS8.3 is still quicker on dual core PC.
However if you use Quad-core should outperform TS8.3 because it uses all 4 cores in spite of calculation inefficiency.

We believe MC 2.2 beta will be out in September. So it will be relatively soon.

glam_100
Posts: 157
Joined: 14 Jun 2006
Has thanked: 2 times
Been thanked: 9 times

Postby glam_100 » 19 Jul 2007

Hi Andrew,

Thanks for the update.

You guys have a great product and I'm sure you can overcome the signal study inefficiency problem. I look forward to your optimized version with Multichart 2.2. If you can get on par or better performance than TS on a per core basis, Multichart will easily become the top strategy development tool available.

I look forward to MultiChart 2.2, and will check back regularly.

Cheers

User avatar
Andrew Kirillov
Posts: 1589
Joined: 28 Jul 2005
Has thanked: 2 times
Been thanked: 31 times
Contact:

Postby Andrew Kirillov » 19 Jul 2007

Please check our forum from time to time. It is possible that we will release a "Performance Boost Service Pack earlier than I expected.

glam_100
Posts: 157
Joined: 14 Jun 2006
Has thanked: 2 times
Been thanked: 9 times

Postby glam_100 » 19 Jul 2007

Thanks Andrew. I eagerly look forward to this performance boost.

User avatar
Andrew Kirillov
Posts: 1589
Joined: 28 Jul 2005
Has thanked: 2 times
Been thanked: 31 times
Contact:

Postby Andrew Kirillov » 20 Jul 2007

I've got the first figures after some improvements our engineers made. Final results should be better in Mc 2.2.

MC is > 2 times faster than TS8.3 on dual core pc. Of course it will be even faster on quad core.

glam_100
Posts: 157
Joined: 14 Jun 2006
Has thanked: 2 times
Been thanked: 9 times

Postby glam_100 » 20 Jul 2007

Wow that's really exciting!

I can't wait till Oct for MC 2.2

Is there any chance for a "performance boost" patch as mentioned above?

glam_100
Posts: 157
Joined: 14 Jun 2006
Has thanked: 2 times
Been thanked: 9 times

Postby glam_100 » 20 Jul 2007

Andrew,

If possible pls consider adding an option to set the number of optimization thread the spawn in case some users would want to leave one core for other work.

The computer would get very unresponsive when all cores are used.

I have no problem with it personally cos I want fastest speed possible but would be nice for other users.

User avatar
Andrew Kirillov
Posts: 1589
Joined: 28 Jul 2005
Has thanked: 2 times
Been thanked: 31 times
Contact:

Postby Andrew Kirillov » 23 Jul 2007

Yes, it is possible to make this optional. We will consider it.

Guest

Postby Guest » 23 Jul 2007

I'm with you Glam, speed is all important, but, couldn't you just set processor affinity to use just 1 core/processor?

Guest

Postby Guest » 30 Jul 2007

Just upgraded from Dual to Quad and to Vista x64. It works very well to set affinity for MCActiveX.exe and Multicharts.exe to use only 3 cores. Benefit of 3 cores crunching optimization and computer remains completely responsive for other tasks.


Return to “MultiCharts”